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[57] ABSTRACT 

A method implemented in an object-based video encoder or 
decoder uses shs^ informatioo that describes the boundary 
of a group of pixels representing an object in a sequence of 
video frames to identify transparent blocks (eg., maaob- 
locks or bloclcs so that coding/decoding of these blocks can 
be skipped. Id the object-based video coding mcdiod, encod- 
ers code sh^ separately from motion and texture, and 
sh{q>e information is available before the encoder/decoder 
codes/decodes texture and motion data. The encoder and 
dccodCT use this shape information to identify transparent 
macroblodcs or blocks so that texture coding and possible 
motion coding can be skipped. This method for transparent 
block skipping reduces coding and decoding operations and 
reduces Ac number of bits needed to store a bitstream 
repo-esenting a compressed video sequence. 

18 Claims, 39 Drawing Sheets 
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TRANSPARENT BLOCK SiOPPING IN 
OBJECT-BASED VIDEO CODING SYSTEMS 

FIELD OF THE INVENTION 

The invention rdates to {socesses for coding video sig- 
nals and, in particular, to an object-based video coding 
process. 

BACKGROUND OF THE INVENTION 

FuU-motion video displays based upon analog video 
signals have long been available in the fonn of television. 
With recent increases in computer processing capabilities 
and affbrdability^ full-motion video displays based upon 
digital video signals are beconoing more widely available. 
Digital video systems can provide significant in^ovements 
over conventional analog video systems in creating, 
modifying, transmitting, storing, and playing full-motion 
video sequences. 

Digital video displays include large numbers of image 
frames that are played or rendered successively at firequen- 
dcs of between 30 and 75 Hz. Each image frame is a still 
image formed from an array of pixels according to the 
display resolution of a particular system. As examples, 
VHS-based systems have display resolutions of 320x480 
pixels, NTSC-based systems have display resolutions of 
720x486 pixels, and high-definition television (HDTV) sys- 
tems under development have display resolutions of 1360x 
1024 pixels. 

The amounts of raw digital information included in video 
sequences are massive. Storage and transmission of these 
amounts of video information is infeasible with conven- 
tional personal computer equipment With reference to a 
digitized form of a relatively low resolution VHS image 
format having a 320x480 pixel resolution, a flU-length 
motion picture of two hours in duration could correspond to 
100 gigabytes of digital video information. By comparison, 
conventional compact optical disks have capacities of about 
0.6 gigabytes, magnetic hard disks have capacities of 1-2 
gigabytes, and compact optical disks under development 
have capacities of up to 8 gigabytes. 

In response to the limitations in storing or transmitting 
such massive amounts of digital video information, various 
video compression standards or processes have been 
cstabUshcd, including MPEG-1, MPEG-2, and H.26X. 
These conventional video compression techniques utilize 
similarities Ixtween successive image frames, referred to as 
temporal or interframe correlation, to provide interframe 
compression in which pixel-based representations of image 
frames are converted to motion representations. In addition, 
the conventional video compression techniques utilize simi- 
larities within image frames, referred to as spatial or 
intraframe conelation, to provide intraframc compression in 
which the motion representations within an image frame are 
further compressed. Intraframe compression is based upon 
conventional processes for compressing still images, such as 
discrete cosine transform (DCT) encoding. 

Although differing in specific implementations, the 
MPEG-1, MPEG-2, and H.26X video compression stan- 
dards are similar in a number of respects. The following 
description of the MPEG-2 video compression standard is 
generally applicable to the others. 

MPEG-2 provides interframe compression and intraframe 
con^xression based upon square blocks or arrays of pixels in 
video images. A video image is divided into transfonnation 
blocks having dimensions of 16x16 pixds. For each trans- 
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formation block TN in an image frame N, a seardi is 
peffonned across the image of a next successive video frame 
N+1 or immediately preceding image frame N-1 (i.e., 
bidirectionaily) to identify the most similar respective trans- 

3 formation blocks TN+1 or TN-1. 

Ideally, and with reference to a search of the next suc- 
cessive image frame, the pixels in transformation blocks TN 
and TN+1 are identical, even if the transfonnation blodcs 
have different positions in thefr respective image frames. 
Under these circumstances, the pixel information in trans- 
formation block TN+1 is redundant to that in transformation 
block TN. Compression is achieved by substituting the 
positional translation between transformation blocks TN and 
TN+1 for the pixel information in transfonnation block 

1^ TN+1. In this simplified example, a single translational 
vector designated (X. Y) for the video information associ- 
ated with each of the 256 pixels in transformation block 
TN+1. 

Frequently, the video information (i.e., pixels) in the 

^ corresponding transformation blocks TN and TN+1 are not 
identical. The difference between them is designated a 
transformation block error E, which often is significant 
Although it is compressed by a conventional compression 
process such as discrete cosine transform (DCT) encoding, 

^ the fransformation block error E is cumbersome and limits 
the extent (ratio) and the accuracy by which video signals 
can be compressed. 

Large transfonnation block errors E ahse in block-based 
video compression methods for several reasons. The block- 

^ based motion estimation represents only translational 
motion between successive image frames. The only change 
between corresponding transformation blocks TO andTO+1 
that can t>e represented are changes in the relative positions 
of the transformation blocks. A disadvantage of such repre- 
sentations is that full-motion video sequences frequently 
include complex motions other than translation, such as 
rotation, magnification and sheer. Representing such com- 
plex motions with simple translational approximations 
results in the significant errors. 

Another aspect of video displays is that they typically 
include multiple image features or objects that change or 
move relative to each other. Objects may be distinct 
characters, articles, or scenery within a video display. With 
respect to a scene in a motion picture, for exaiiq)le, each of 
the characters (i.e., actors) and articles (i.e., props) in the 
scene could be a different object. 

The relative motion between objects in a video sequence 
is another source of significant transformation block errors 

50 E in conventional video compression processes. Due to the 
regular configuration and size of the transfonnation blocks, 
many of them encompass portions of different objects. 
Relative motion between the objects during successive 
image frames can result in extremely low correlation (i.e.» 

55 high-transformation errors E) between corresponding trans- 
formation blocks. Similarly, the appearance of portions of 
objects io successive image frames (e.g., when a character 
turns) also introduces high-transformation errors E. 

Conventional video compression ructhods appear to be 

GO inherently limited due to the size of transformatioQ errors E. 
With the increased demand for digital video display 
capabilities, improved digital video concession processes 
are required. 

SUMMARY OF THE INVENTION 

The invention provides a method for reducing overhead 
during the encoding and decoding of video "objects" in an 
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object-based video encoder. An object is a group of pixels in FIG. is a generalized fiinctioiial block diagram of a 

a video firame used to display something that behaves as a video compression encoder process for conqjressing digi- 

physical entity. In particular, this entity prcfo-ably demon- tizcd video signals representing display motion in video 

strates relatively rigid body motion and color invariance, but sequences of multiple inoage frames. FIG. 3B is a functional 

this is not an absolute requirement. Object-based coding is 5 block diagram of a master object encoder process, 

a region-based coding sdieme (as opposed to a block based FIG. 4 is a functional block diagram of an object seg- 

coding scheme) where the regions are defined by the shapes mentation process for segmenting selected objects from an 

of the objects. The method of the invention reduces coding image firame of a video sequence, 

overhead and the number of bits needed to code objects in FIG. 5A is simplified representation of display screen of 

a sequence of video frames by using shape information to jg the video display device of FIG. 2A, and FIG. 5B is an 

identify transparent transformation blocks around an object enlarged representation of a portion of the display screen of 

and then skipping encodin^decoding of these blocks. FIG. 5A. 

In an object-based video encoder or decoder designed FIG. 6 is a functional block diagram of a polygon match 

according to the invention, shape information is available process for determining a motion vcctcr for corresponding 

independent of motion estimation and texture information. 15 of pixels in corresponding objects In successive image 

As such, the method of the invention can use the shape frames. 

information to identify transparent transfonnation blocks FIGS. 7 A and 7B are simplified representations of a 

and skip texture and possibly motion coding and decoding display screen showing two successive image frames with 

fcH- these blocks. An encoder enq)loying this method evalu- COTesponding (Ejects. 

atcs the shape of an object to determine whether a given 20 * ^® * functional block diagram of an alternative 

block is transparent, ie, covered by the object If the block ^^^^ correlation process. 

is transparent, die encoder can skip texture coding for inter ^ schematic representation of a first pixel block 

and intra frame blocks. The encoder can also skip coding of identifying corresponding pixels in different image 

rooaon estimation data, such as motion vectors or transfor- ^^^^ schematic representation of an array of 

mation coefficients for inta frame blocks. Similarly, the 23 P"^cl5c<>^Pon<5fgto ascarcharcain a^^^^ 

decoder can use decoded shape informaUon to identify "^^^ correspondmg pixels are sought FIGS. 9C-9G are 

^ * Li I J 1 ■ * 1 ^ J J. X schematic representations of the first pixel block beine 

tomspareat blocks and slop torture or motion decoding for ^ ,B .^^^ 

these blocks. corresponding pixels. 

Tht method of the invention appEes to tranrformation piG. lOA is a schematic representation of a second pixel 

Mocks as weU as smaUer blocks inside a transfamation 30 block used for identifying coiresponding pixels in different 

Mock (sub-transformation blocks). TTje objects in an object jniage frames. FIGS. lOB-lOF are schematic representa- 

based coding scheme have an associated bounding region, (ions of the second pixel block being seamed across the 

typicaUyaboundingrectanglcthatencloses the boundary or pi^d anay of HG. 9B to identify corresponding pixels, 

"shve" of the objert^To encode inotion and texture data for piQ. ^ » sdiemaUc representation of a third pixel 

an object, the encote divides the boundmg region into 35 block used for identifying coiresponding pixels in dlffaent 

fransfonnabon blocks and encodes the object s motion and f,^^ pigs. UB-UF are schematic representations 

texture dau for these block. Li some implementations, „f j^^, y^ek being scanned aaoss the pixel array 

transformation blocks are further divided into smaller pjQ r ^ 

blocks, which we refer to as subtransforraation blocks. One ct^' -J . , ui^v » u- 

-f., ^ jt^^r ^Lii fiCj. 12 is a function block diagram of a multi- 

cxanmle of transformation and sub-lransformation blocks 40j- . i*__r ^j^t^^.ij 

TV 1^ . I i_i 1 J o o I 1 1.1 1 dimensional transformation method that mcludcs cencralmg 

are 16x16 pixel macroblocks and 8x8 pixel blocks. ^ . ... «*. • ^ ^ - 

• ._, 1 J . . -i. , a mapping between objects in first and second successive 

However,thcsizcoftheseblockscanvary and IS not critical . i*- • * 

. ^ image frames and quantitizing the mapping for transmission 

e invcn on. . , . ^ or storage. 

The mvenuon provides a number of advantages. One „ ^ ^ representation of a display screen 

significant. «lvantage is that transparen^ block stopping 45 u„agVframe of HG. 7B for i^ses of 

«vcs operations^ad of encoding transparent blodta J„ ^v. muM-dimensional tranrformatioVTthod of 

they are merely skipped. This improves performance in both ^ * 

the encoder and decoder. In addition to saving operations, ^, 

the method reduces the number of bits required to encode , }^ .^^8?^ simplified representation showing 

object-based video. Rather than coding transparent portions 30 selected pixels of a ^sformaton block used m the 

of a bounding region with a constant, sudi as zero values for ^"T^^'i?^ a^ fransfonnation coefficients determined 

aU transparent pixels, transparent blocks can be skipped method of FIO. 12. 

without sending any additional information. No additional ™- * functional block diagram of a transfonnation 

information is necessary because shape information is avail- optimization method utiUzcd in an alternative embodi- 

ablc independently and can be used to identify tran^arent 55 ^^^^ multi-dimensional transformation method of 



blocks. 



FIG. 12. 



These and additional features and advantages of the ^ simplified fragmentary representotion of a 

invention wiU become more apparent from the following dispUy screen showing the image frame of HG. 7B for 
detailed description and accompanying drawing. purposes of illustrating the transfcHmation block pptimiza- 

60 ^<>^B^^CK^of ^^G. 15. 

BRIEF DESCRffnON OF THE DRAWINGS piGS. 17A and 17B are a functional block diagram of a 

FIG. 1 is a block diagram of a computer system that may preconqiression extrapolation method for extrapolating 
be used to implement a method and apparatus embodying image features of arbitrary configuration to a predefined 
the invention. configuration to facilitate compression. 

FIGS. 2A and 2B are sin^lified representations of a 6S FIGS. 18A-18D are representations of a display screen on 
display screen of a video display device showing two which a sirxqde object is rendered to show various aspects of 
successive image frames corresponding to a video signaL the extrapolation method of FIG. 14. 
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FIGS. 19A and 19B are functional block diagrams of an FIG. 41 is a flow diagram Illustrating a method for 

cDCodcr method and a decoder method, respectively, skipping transparent macroblocks in an object-based video 

employing a Laplacian pyramid encoder method. decoder. 

FIGS. 20A-20D are simpliiied r^jreseatations of the FIG. 42 is a flow diagram illustrating transparent block 
color component values of an arbitrary set or array of pixels s skipping for partially covered macroblocks in an object- 
processed according, to the encoder process of FIG. 19A. based video encoder. 

FIG. 21 is a functional block diagram of a motion vector FIG. 43 is a flow diagram illustrating transparent block 

encoding process. skipping for partially covered macroblocks in an object- 

FIG. 22* is a functional block diagram of an alternative video decoder, 

quanlized object encoder-decoder process. '° DETAILED DBSOOFnON 

FIG. 23A is a generalized functional block diagram of a 

video con^ssion decoder process matched to the encoder Refening to FIG. 1, an operating environment for the 

process of HG. 3. FIQ, 23B is a functional diagram of a preferredcmbodimentof the present invention is a computer 

master object decoder jffocess. system 20, cither of a general purpose or a dedicated type, 

FIG. 24Ais a diagrammatic rcFesentation of a conven- ^lat comprises at least one high speed proc^^^^ 

tional chain code foimat FIG. 24B is a simplified reprcsen- 22, m conjunction xi^th a nu^on^ 

tation of an exemplary contour for processing with the diain ^^J^ ^"^'^t ^' ^^'^ 

code format of na W nected by a bus structure 30. 

na25AisafunctionalblockdiagramofachaiD coding 20 Tihe muffed CPU 22 is of famiHar design 

an ALU 32 for performing conq)utations, a colicctioD of 

T-,^ J. ^ . ^- X u . J registers 34 for temporary storage of data and instructions, 

nG.2SBisadxagrainmt»crepreseQtationofachaiiicode and a control umt36for controlling operation of the syst^ 

roimaL ... 20. CPU 22 may be a processor having any of a variety of 

FIG. 25C is a diagrammatic representation of special case architectures including Alpha from Digital, MIPS from 

chain code modifications used in the process of FIG. 25 A. ^ j^^g Technology, NEC, IDT, Siemens, and others, x86 

FIG. 26 is a functional block diagram of a sprite gener- from Intel and others, including Cyrix, AMD, and Nexgen, 

ating or encoding process. and the PowerPC from IBM and Motorola. 

FIGS. 27A and 27B are respective first and second objects Tbe memory system 24 includes main memory 38 and 

defined by bitm;^s and showing grids of triangles superim- ^ secondary storage 40. Illustrated main memory 38 takes the 

posed over the objects in accordance with the process of fonn of 16 megabytes of semiconductor RAM memory. 

FIG. 26. Secondary storage 40 takes the form of long term storage, 

FIG. 28 is a functional block diagram of a sprite decoding such as ROM, optical or magnetic disks, flash memory, or 

process corresponding to the encoding ^ocess of FIG. 26. t^. Those skilled in the art will appreciate that memory 

FIG. 29 is a diagrammatic rqjresentation of an exeii^)lary 35 system 24 may con^irise many other alternative conqx)- 

simple arbitrary binary solid shape corresponding to a mask nents. The input and output devices 26, 28 are also fanailiar. 

of an object included in a frame of a video sequence. The input device 26 can comprise a keyboard, a noouse, a 

FIG. 30A is a diagrammatic representation of an exem- physical transducer (e.g., a microphone), etc The output 

plary general binary arbitrary shapes coircsponding to a device 28 can comprise a display, a printer, a transducer (e.g. 

mask of a complex object in a frame of a video sequence. 40 * speaker), etc. Some devices, such as a network interface or 

HGS. 30B-30D arc diagrammatic representations of die ^ modem, can be used as input and/or output devices, 

general binary arbitrary sh^)es. As is familiar to those skilled in the art, the computer 

FIG. 31 is a functional block diagram of a hierarchical system 20 further includes an operating system and at least 

dccon^sition and encoding process capable of accurately application jffogram. The operating system is the set of 

representing general binary aibitraiy shapes of the type 45 software which controls the conqwtcr system's operation 

shown in FIG 30A ^ allocation of resources. The application program is 

no. 32 is a fiinctional block diagram of an encoding ^et of software that performs a task desired by the user, 

process for representing non-binary object information such ^5 computer resources rnade available through 

as object transparency data. operatmg system. Both are resident m the illustrated 

FIG. 33 is a block diagram iUustrating the structure of an ^0 memory system 24 ^ 

object-based video encoder. ^ accordance with the practices of persons skilled in the 

RG. 34 is a block diagram illustrating the structure of an f '^^u^'^!''^ pt.gramming the present invention is 

object-based video decoder. dcscnbed below with refoence to symboUc r^sentations 

^- „, ^ , . - ... J. .J ^ of operations that are peifomed by computer system 20, 

FIG. 35 illustrates how a frame of video can be divided „ „«i„„ r.**.™.:,* « 

55 unless mdicated otherwise. Such operations are sometimes 

into the objects m the frame. ^^^^^^ ^ ^ computer-executed. It will be appred- 

FIGS. 36 is a general block diagram illustrating parts of operations whidi are symboUcally rq>resented 

an object-based video encoder. include the manipulation by CPU 22 of electrical signals 

FIGS. 37 is a general block diagram illustrating parts of representing data bits and the maintenance of data hits at 

an object-based video decoder. ^ memory locations in memory system 24, as well as otho- 

FIG. 38 is a block diagram illustrating an implementation processing of signals. The memory locations where data bits 

of an object-based video encoder. are maintained are physical locations that have particular 

FIG. 39 is a block diagram illustrating an implementation electrical, magnetic, or optical properties corresponding to 

of an object-based video coding method. the data bits. 

FIG. 40 is a flow diagram illustrating a method ioiple- 69 FIGS. 2A and 2B are simplified representations of a 

mented in an object-based video encoder to skip transparent display screen 50 of a video display device 52 (e.g., a 

macroblocks. television or a conputer monitor) showing two successive 
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image frames 54a and 54£r of a video image sequence 
represented electrooicaUy by a codiespoiiding video signal. 
Video signals may be in any of a variety of video signal 
fonnats including analog television video fonnats such as 
NTSC, PAL* and SECAM, and pixelated or digitized video 
signal formats typically used in computer displays* such as 
VGA, CGA, and EGA. Preferably, the video signals corre- 
sponding to image frames are of a digitized video signal 
facmat, either as originally generated or by conversion from 
an analog video signal format, as is Imown In the art 

Image frames 54a and S4b each include a rectangular 
solid image feature 56 and a pyramid image feature 58 that 
are positioned over a baclEground 60. Image features 56 and 
58 in image frames 54a and S4b have different appearances 
because different parts are obscured and shown. For pur- 
poses of the following description, the particular form of an 
image feature in an image frame is rcfezrcd to as an object 
or, alternatively, a mask* Accordingly, rectangular solid 
image feature 56 is shown as rectangular solid objects 56a 
and 56^ in respective image frames 54a and S4b, and 
pyramid image feature 58 is shown as pyramid objects 58a 
and 58^ in respective image frames 54a and 546. 

pyramid image feature 58 is shown with the same position 
and orientation in image frames 54a and 546 and would 
"appear" to be motionless when shown in the video 
sequence. Rectangular solid 56 is shown in frames 54a and 
546 with a different orientation and position relative to 
pyramid 58 and would "appear" to be moving and rotating 
relative to pyramid 58 when shown in the video sequence. 
These appearances of image features 58 and 60 are figura- 
tive and exaggerated. The image frames of a video sequence 
typically are displayed at rates in the range of 30-80 Hz. 
Human perception of video motion typically requires more 
than two image frames. Image frames 54a and 546 provide, 
therefore, a sinq>lified representation of a conventional 
video sequence for purposes of illustrating the present 
invention. Moreover, it will be appreciated that the present 
invention is in no way limited to such simplified video 
images, image features, or sequences and, to the contrary, is 
applicable to video images and sequences of arbitrary com- 
plexity. 

Video Compression Encoder Process Overview 

FIG. 3A is a generalized functional block diagram of a 
video oomprcssioo encoder process 64 for compressing 
digitized video signals representing display motion in video 
sequences of multiple image frames. Compression of video 
informition (Le., video sequences or signals) can provide 
economical storage and transmission of digital video infor- 
mation in applications that include, for example, interactive 
or digital television and multinoedia computer applications. 
For purposes of brevity, the reference numerals assigned to 
function blocks of encoder process 64 arc used interchange- 
ably in reference to the results generated by the function 
blocks. 

Conventional video compression techniques utilize simi- 
larities 'between successive image frames, refetred to as 
temporal or intetframe correlation, to provide interframe 
compression in which pixel-based rq>resentations of image 
frames are converted to motion representations. In addition, 
conventional video compression techniques utilize similari- 
ties within image frames, referred to as spatial or intraframe 
condation, to provide intraframe conq)ression in which the 
motion rqirescntations within an image frame are further 
compressed. 

In such conventional video compression techniques, 
including MPEG-1, MPEG-2, and H.26X, the temporal and 
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spatial cQirelations are determined relative to simple trans- 
lations of fixed, regular (e.g., square) arrays of pixels. Mdeo 
information commonly Includes, however, arbitrary video 
motion that cannot be r^iresented accurately by translatiag 
5 square arrays of pixds. As a consequence, conventional 
video compression techniques typically indude significant 
error conq)onents tiiat limit the compression rate and accu- 
racy. 

In contrast, encoder process 64 utilizes object-based video 

10 con^nression to improve the accuracy and versatility of 
encoding inter&ame motion and intraframe image features. 
Encoder process 64 compresses video information rdative 
to objects of arbitrary configurations, rather than fixed, 
regular airays pixels. This reduces the oror components 

13 and thereby in^roves the compression effidency and accu- 
racy. As another benefit, object-based video compression 
provides interactive video editing capabilities for processing 
compressed video Infcamatlon. 
Referring to FIG. 3A* function block 66 indicates that 

^ user-defined objects within image frames of a video 
sequence are segmented from other objects within the image 
frames. The objects may be of arbitrary configuration and 
preferably represent distinct image features in a display 
image. Segmentation indudes identifying the pixds in the 

23 image frames corresponding to the objects. The user-defined 
objects are defined in each of the image frames in the video 
sequence. In HGS. 2A and 2B, for example, rectangular 
solid objects 56a and 566 and pyramid objects 58a and 586 
are s^iarately segmented. 

^ The segmented objects are represented by binary or 
multi-bit (e.g., 8-bit) "a^)hachannd" masks of the objects. 
The object masks indicate the size, configuration, and posi- 
tion of an object on a pixd-by-pixel basis. For purposes of 
simplidty, the following description is directed to binary 
masks in which each pixel of the object is represented by a 
single binary bit rather than the typical 24-bits (i.e., 8 bits for 
each of three color component values). Multi-bit (e.g., 8-lrit) 
masks also have been used. 

^ Function block 68 indicates that 'feature points'* of each 
object are defined by a user. Feature points pr^erably are 
distinctive features or aspects of the object For example, 
comers 70a-70c and comers 72a-72c could be defined by a 
user as feature points of rectangular solid 56 and pyramid 
58, respectively. The pixels corre^nding to each object 
mask and its feature points in each image frame are stored 
in an object database included in memory system 24. 

Function block 74 indicates that changes in the positions 
of feature points in successive image frames are identified 
and trajectories determined for the feature points between 
successive image fi^mes. The trajectories represent ^c 
direction and extent of movement of the feature points. 
Function block 76 indicates that trajectories of the feature 
points in the object between prior frame N-1 and current 

33 frame N also is retrieved from the object data base. 

Function block 78 indicates that a sparse motion trans- 
fonnation is determined for the object between prior frame 
N-1 and current frame N. The sparse motion transfonnation 
is based upon the feature point trajectories between frames 

^ N-1 and N. The sparse motion transformation provides an 
approximation of the diange of die object between prior 
frame N-1 and cuirent frame N. 

Function block 80 indicates that a mask of an object In a 
cuiient frame N is retrieved from the object data base in 

65 memory system 24. 

Function block 90 indicates that a quantized master object 
or "sprite** is formed from the objects or masks 66 coire- 
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sponding to an image feature in an image frame sequence Function block 112 indicates that estimated eiror 110 is 
and feature point trajectories 74. The master object prefer- con^>rcsscd or "coded" by a conventional "lossy*" still image 
ably includes all of the aspects or features of an object as it concession method such as lattice subband (wavelet) corn- 
is represented in multiple frames. reference to FIGS. pression or encoding as described in Multirate Systems and 
2A and 2B, for example, rectangular solid 56 in frame 54b 5 Filter Banks by Vaidyanathao, FFR Ptentice-Hall, Inc, 
includes a side 786 not shown in fi^e S4a, Similarly, Englewood Cliffs, KJ., (1993) or discrete cosine transform 
rectangular solid 56 includes a side 78a in frame S4a not (DCD encoding as descnbed in JPEG: SUU Image Data 
shown in frame 546. The master object for rectangular solid Compression Standard by Pennebaker et aL, Van Nostrand 
56 includes both sides 78a and 786. Rcinhold, New York (1993). 

Sparse motion transformation 78 frequently will not pro- 1° As is known in the art, 'loss/* compression methods 
vide a complete representation of the diange in the object introduce some data distortion to provide increased data 
between frames N-1 and N. For example, an object in a prior compression. TTie data distortion refers to variations 
frame N-1, such as rectangular object 54a, mig^t not include between the original data before compression and the data 
all the features of the object in the current frame N, such as resulting after compression and decarajK-ession. For pur- 
side 786 of rectangular object 546. poses of illustration below, the conq)ression or encoding of 

To improve the accuracy of the transformation, therefore, function block 102 is presumed to be wavelet encoding, 

an intersection of the masks of the object in prior frame N-1 Function block 114 indicates that the wavelet encoded 

and cunent frame N is determined, sudi as by a logical AND estimated etror from function block 112 is further oom- 

function as is known in the art The mask of ttie object in the pressed or "coded** by a conventional "lossless" still image 

current frame N is subtracted from the resulting intersection ^ compression method to form compressed data 116. A pre^ 

to identify any portions or features of the object in the f erred conventional "lossless" still image compression 

current frame N not included in the object in the prior frame method is entropy encoding as described in JPEG: Still 

N-1 (e.g., side 786 of rectangular object 546, as described Image Data Compression Standard by Pennebaker et al. As 

above). The newly identified portions of the object are is known in the art, "lossless** compression methods intio- 

inooiporated into master object 90 so that it includes a duce no data distortion. 

conq)lete representation of the object in frames N-1 and N. An error feedback loop 118 utilizes the wavelet encoded 

Function Mock 96 indicates that a quantized form of an estimated error from function block 112 for the object in 

object 98 in a prior frame N-1 (e.g., rectangular solid object frame N to obtain a prior quantized object for succeeding 

56a in image frame 54a) is transformed by a dense motion frame N+l. As an initial step in feedback loop 118, function 

transformation to provide a predicted form of the object 102 block 120 indicates tiiat the wavelet encoded estimated error 

in a current frame N (e.g., rectangular solid object 566 in from function block 112 is inverse wavelet coded, or wavelet 

image frame 546). TTiis transformation provides object- decoded, to form a quantized error 122 for the object in 

based interfr*ame compression. image frame N. 

The dense motion transformation preferably includes 35 The effect of successively encoding and decoding esti- 

deteimining an affine transformation between quantized mated error HO by a lossy still image compression method 

prior object 98 in frame N-1 and the object in the current is to omit from quantized ecror 122 video information that is 

frame N and applying the affine transfonnation to quantized generally imperceptible by viewers. This information typi- 

prior object 98. The preferred affine transformation is rep- cally is of higher firqucndcs. As a result, omitting such 

resented by flfflnft transformation coefficients 104 and is 4^ higher frequency con^ionents typically can provide image 

capable of describing translation, rotation, magnification, con^>rcssion of up to about 200% with only minimal deg- 

and shear. The affine transformation is determined frx>m a radation of image quality. 

dense motion estimation, preferably including a pixel-by- Function block 124 indicates that quantized error 122 and 

pixel mapping, between prior quantized object 5^ and the predicted object 102, both for image fr^e N, are added 

object in the current frame N. 45 together to form a quantized object 126 for image frame N. 

Predicted current object 102 is represented by quantized After a timing coordination delay 128, quantized object 126 

prior object 98, as modified by dense motion transformation becomes quantized prior object 98 and is used as the basis 

96, and is enable of representing relatively complex for processing the corresponding object in image frame N+1. 

motion, together with any new image aspects obtained from Encoder process 64 utilizes the ten^oral correlation of 

master object 90. Such object-based representations are 50 conesponding objects in successive image fr^es to obtain 

relatively accurate because the perc^tual and spatial con- in^>roved intecfraroc compression, and also utilizes the 

tinuity associated with objects eliminates errors arising from spatial correlation within objects to obtain accurate and 

the typically changing relationships between different efficient intrafrmne compression. For the intcrframc 

objects in ctifferent image frames. Moreover, the object- compression, motion estimation and compensation are per- 

based representations allow a user to represent different 55 formed so that an object defined in one frame can be 

objects with different levels of resolution to optimize the estimated in a successive frame. The motion-based estima- 

rclative efficiency and accuracy for representing objects of tion of the object in the successive frame requires signifi- 

varying con^lexity. cantly less information than a conventional block-based 

Function block 106 indicates that for image frame N, representation of the object For the intrafiamc compression, 

predicted current object 102 is subtracted from original 60 an estimated error signal for each object is compressed to 

object 108 for current frame N to determine an estimated utilize the spatial correlation of the object within a frame and 

error 110 in predicted object 102. Estimated error 110 is a to allow cfifferent objects to be represented al different 

conqncssed rqircscntation of current object 108 in image resolutions. Feedback loop 118 allows objects in subsequent 

frame N relative to quantized prior object 98. More frames to be predicted from fiiUy decompressed objects, 

specifically, current object 108 may be decoded or rccon- 65 thereby preventing accumulation of estimation error, 

structed ficcm estimated error 110 and quantized prior object Encoder process 64 provides as an output a confessed or 

98. encoded representation of a digitized video signal nprcsent- 
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ing display motion in video sequences of multiple image 
frames. Hie compressed ox encoded representation indudes 
object masks 66, feature points 68, affine transfonn coe& 
dents 104, and compressed error data 116. Tht encoded 
representation may be stored or transmitted, according to the 
particular application in uliich the video infonnation is 
used 

FIG. 3B is a functional block diagram of a master object 
encoder process 130 for encoding or confessing master 
object 90. Function block 132 indicates diat master object 90 
is conquessed or coded by a conventional *los5y" still image 
concession method such as lattice subband (wavelet) com- 
pression or discrete cosine transform (DCT) encoding. 
E^cfcrably, function block 132 employs wavdct encoding. 

Function block 134 indicates that the wavelet encoded 
master object from function block 132 is further compressed 
or coded by a conventional ^lossless" still image compres- 
sion method to f€xm compressed master object data 136. A 
preferred conventional lossless still image compression 
method is entropy encoding. 

EncodCT process 130 provides as an output compressed 
master object 136. Together with the compressed or encoded 
representations provided by encoder process 64, conopressed 
master object 136 may be decompressed or decoded after 
storage or transnussion to obtain a video sequence of 
multiple image frames. 

Bnooder process 64 is described with reference to encod- 
ing video information corresponding to a single object 
within an image frame. As shown in FIGS. 2A and 2B and 
indicated above, encoder process 64 is performed separately 
for each of the objects (e.g., objects 56 and 58 of FIGS. 2A 
and 2B) in an image frame. Moreover, many video images 
indude a background over whidi arbitrary numbers of 
image features or objects are rendered. Preferably, the 
background is processed as an object after all user- 
designated objects are processed. 

Processing of tiie objects in an image frame requires that 
the objects be separately identiiied. Preferably, encoder 
process 64 is af^cd to the objects of an image frame 
(beginning with the forward-most object or . objects and 
proceeding suocessivdy to the back-most object (e.g., the 
background). The con^siting of the encoded objects into a 
video image prefCTably pnx:eeds from the rear-most object 
(e.g., the background) and proceeds successivdy to the 
foiward-most object (e.g., rectangular solid 56 in FIGS. 2A 
and 2B). The layering of encoding objects may be commu- 
nicated as distinct layering data associated with the objects 
of an image frame or, altemativdy, by transmitting or 
obtaining the encoded objects in a sequence coiresponding 
to the layering or conq>ositing sequence. 

Object Segmentation and Tracking 

In a preferred embodiment, the segmentation of objects 
within image frames referred to in function block 66 allows 
interactive segmentation by users. This object segmentation 
technique provides improved accuracy in segmenting 
objects and is relatively fast and provides users with optimal 
flexibility in defining objects to be segmented. 

FIG. 4 is a functional block diagram of an object seg- 
mentation process 140 for segmenting sdected objects from 
an image frame of a video sequence. Object segmentation 
according to process 140 provides a perceptual grouping of 
objects that is accurate and quick and easy for users to 
define. 

FIG. 5A is simfWed rqiresentation of display screen 50 
of video display device 52 showing image frame 54a and the 
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segmentation of rectangular solid object 56a. In its render- 
ing on display screen 50, rectangular solid object 56a 
indudes an object perimeter 142 (shown spaced apart from 
object 56a for darity) that bounds an object interior 144. 

3 Object interior 144 refers to the outline of object 56a on 
display screen 50 and in general may correspond to an inner 
surface or, as shown, an outer surface of the image feature. 
FIG. 5B is an enlarged representation of a portion of display 
screen 50 showing the semi-automatic segmentation of 
rectangular solid object 56a. The following descdption is 
made with specific reference to rectangular solid object 56a, 
but is similarly applicable to each object to be segmented 
from an image frame. 

Function block 146 indicates that a user forms within 
object interior 144 an interior outline 148 of object perimeter 
142. The user preferably forms interior outline 148 with a 
conventional pointer or cursor control device, such as a 
mouse or trackball. Interior outline 148 is formed within a 
nominal distance 150 from object perimeter 142. Nominal 

^ distance 150 is selected a user to be suflSdentiy large that 
the user can form interior outline 148 rdativdy quickly 
within nominal distance 150 of perimeter 142. Nominal 
distance 150 corresponds, for example, to between about 4 
and 10 pixds. 

Function block 146 is performed in connection with a key 
frame of a video sequence. WiUi reference to a scene in a 
conventional motion picture, for example, the key frame 
could be the first frame of the multiple frames in a scene. 
The partidpation of the user in this function renders object 
segmentation process 140 semi-automatic, but signiiicantiy 
increases the accuracy and flexibility with which objects are 
segmented. Other than for the key frame, objects in subse- 
quent image frames are segmented automatically as 
described below in greater detail. 
35 Function block 152 indicates that interior outline 148 is 
expanded automatically to form an exterior outline 156. The 
formation of exterior outline 156 is perfonned as a rdativdy 
simple image magnification of outline 148 so that exterior 
outline 156 is a user-defined number of pixels from interior 
outline 148. Prcfaably, the distance between interior outline 
148 and exterior outline 156 is approximately twice distance 
150. 

Function block 158 indicates that pixels between interior 
outline 148 and exterior outline 156 are classified according 

45 to predefined attributes as to whether they are within object 
interior 144, thereby to identify automatically object perim- 
eter 142 and a corresponding mask 80 of the type described 
with reference to FIG. 3A. Preferably, tiie image attributes 
indude pixd color and position, but dther attribute could be 

50 used alone or witti other attributes. 

In tiie preferred embodiment, each of the pixels in interior 
outline 148 and exterior outline 156 defines a "cluster 
center^ r^resented as a five-dimensional vector in the form 
of (r, g, b, X, y). The terms r, g, and b correspond to the 

55 respective red, green, and blue color components associated 
with each of the pixds, and the terms x and y correspond to 
the pixel locations. The m-number of cluster center vectors 
corresponding to pixels in interior outline 148 are denoted as 
{lo, Ij, . . . , Im-i}» and the n-number of duster center vectors 

60 corresponding pixels in exterior outline 156 are denoted as 
{Oo, 0„ . . . , 0„_j}. 

Pixels between the duster center vectors 1^ and Oj are 
classified by identifying the vector to whidi each pixel is 
closest in the five-dimensional vector space. For each pixel, 

65 the absolute distance d^ and d^- to each of respective duster 
center vectors If and Oj is computed according to the 
following equations: 



05/13/2004, EAST Version: 1.4.1 



5,7- 

13 

in which w^^^ and are weighting factors for the 

respective color and pixel positioD iofcsiDation. WeightiDg 
factors and are of values having a sum of 1 and 
otherwise selectable by a user. Preferably, weighting factors 
and w^^ are of an equal value of 0.5. Each pixel is 
associated with object intedor 144 or exterior according to 
the minimum five-dimensional distance to one of the cluster 
center vectors I,- and Oy. 

Function block 162 indicates that a user selects at least 
two, and pr^erable more (e.g. 4 to 6), feature points in each 
object of an initial or key frame. Preferably, the feature 
points arc relatively distinctive aspects of the object With 
reference to rectangular solid image feature 56, for examjHe, 
comers 70a-70c could be selected as feature points. 

Function block 164 indicates that a block 166 of multiple 
pixels centered about each selected feature point (e.g., 
comers 70a-70c) is defined and matched to a corresponding 
block in a subsequent image frame (e.g., the next successive 
image frame). Pixel block 166 is user defined, but preferably 
includes a 32x32 pixel array that includes only pixels within 
image interior 144. Any pixels 168 (indicated by cross- 
hatching) of pixel block 166 falling outside object interior 
144 as determined by function block 158 (e.g., comers 70b 
and 70c) are omitted. Pixel blocks 166 arc matched to the 
corresponding pixel blocks in the next image frame accord- 
ing to a minimum absolute error identified by a conventional 
block match process or a polygon match process, as 
described below in greater detail 

Function block 170 indicates that a sparse motion trans- 
formation of an object is detomined from the corresponding 
feature points in two successive image frames. Function 
block 172 indicates that mask 80 of the current image frame 
is transformed according to die sparse motion transforma- 
tion to provide an estimation of the mask 80 for the next 
image frame. Any feature point in a current frame not 
identified in a successive image frame is disregarded. 

Function block 174 indicates that the resulting estimation 
of mask 80 for the next image fr^me is delayed by one 
frame, and functions as an outUne 176 for a next successive 
cycle. Similarly, function block 178 indicates that the cor- 
responding feature points also are delayed by one frame, and 
utilized as the initial feature points 180 for the next succes- 
sive frame. 

Polygon Match Mefliod 

FIG. 6 is a functional block diagram of a polygon match 
process 200 for determining a motion vector for each 
corresponding pair of pixels in successive image frames. 
Such a dense motion vector determination provides the basis 
for determining the dense motion transfomiations 96 of FIG. 
3A. 

Polygon match process 200 is capable of determining 
extensive motion b^een successive image frames like the 
conventional block match process. In contrast to the con- 
ventional block match process, however, polygon match 
process 200 maintains its accuracy for pixels located near or 
at an object perimeter and generates significantly less error. 
A preferred embodiment of polygon match method 200 has 
improved computational ef&dency. 

Polygon block method 200 is described with reference to 
FIGS. 7A and 7B, which are simplified representations of 
display screen 50 showing two successive image frames 
202a and 202b in which an image feature 204 is rendered as 
objects 204a and 204A, respectively. 
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Function block 206 indicates that objects 204a and 204i^ 
for image frames 202a and 202^ are identified and seg- 
mented by, for exan^le, object segmentation method 140. 
. Function block 208 indicates that dimensions are dcter- 
5 mined for a pixel block 210b (e.g., 15x15 pixels) to be 
applied to object 204b and a search area 212 about object 
204a. Pixel block 2l0b defines a region abcAit each pixel in 
object 204b for which region a corresponding pixel block 
210a is identified in object 204a. Search area 212 establishes 
10 a region within which corresponding pixel block 210a is 
sought Preferably, pixel block 210^ and search area 212 are 
right regular arrays of pixels and of sizes defined by the user. 

Function block 214 indicates that ao initial pixel 216 in 
object 2Mb is identified and designated the current pixel 
15 Initial pixel 216 may be defined by any of a variety of 
criteria such as, for example, the pixel at the location of 
greatest vertical extent and minimum horizontal extent With 
the pixels on display screen 50 arranged according to a 
coordinate axis 220 as shown, initial pixel 216 may be 
^ rqjiesented as the pixel of object 214& having a maximum 
y-coordinate value and a minimum x-coordinate value. 

Function block 222 indicates that pixel block 2l0b is 
centered at and extends about the current pixel. 

Function block 224 represents an inquiry as to whether 
pixel block 2106 includes pixels that are not included in 
object 2046 (e.g., pixels 226 shown by cross-hatching in 
FIG. 7B). This inquiry is made with reference to the objects 
identified according to function block 206. Whenever pixels 
^ within pixel block 2106 positioned at the current pixel fall 
outside object 2046, function block 224 proceeds to function 
block 228 and otherwise proceeds to function block 232. 

Function block 228 indicates that pixcb of pixel block 
2106 falling outside object 2046 (e.g., pixels 226) are 
3^ omitted from the region defined by pixel block 2106 so that 
it includes only pixels within object 2046. As a result, pixel 
block 2106 defines a region that typically would be of a 
polygonal shape more con^lex than the originally defined 
square or rectangular region. 

Fiuction block 232 indicates that a pixel in object 204a is 
identified as corresponding to the current pixel in object 
2046. The pixel in object 204a is referred to as the prior 
carresponding pixel Heferably, the prior corresponding 
pixel is identified by forming a pixel block 210a about each 
43 pixel in search area 212 and determining a correlation 
between the pixel block 210a and pixel block 2106 about the 
current pixel in object 2046. Each correlation between pixel 
blocks 210a and 2106 may be determined, for example, a 
means absolute error. The prior corresponding pixel is 
50 identified by identifying the pixel block 210a in search area 
212 for which the mean absolute error relative to pixel block 
2106 is minimized. A mean absolute error E for a pixel block 
210a relative to pixel block 2106 may be determined as: 

in which the terms r^^', g^\ and b^ ' correspond to the 
respective red, green, and blue color conq)onents associated 
60 with each of the pixels in pixel block 2106 and the terms r^', 
gy', and b^' correspond to the respective red, green, and blue 
color con^onents associated with each of the pixels in pixel 
block 210a. 

As set forth above, the sununations for the mean absolute 
65 error E imply pixel blocks having pixel arrays having mxn 
pixel dimensions. Pixel blocks 2106 of polygonal configu- 
ration are accommodated relatively singly by, for cjcamplc, 
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defining zero values for the cdor componeots of all p}j.c\s 
outside polygonal pixel blocks 210h 

Function block 234 indicates that a motion vector MV 
between each pixel in object 2Mb and the CQtresponding 
prior pixel in c^ject 204a is detcnnined. A motioo vector is 
defined as the difference between the locations of the pixel 
in object 204b and the corresponding prior pixel in object 
204a: MV^tx.—Xjt'l, ly^-y/), in which ttic tenns x, and 
correspond to the respective x- and y-coordinatc positions of 
the pixel in pixel block 2lQb , and the terms x^' and y^' 
c(»respond to the respective x- and y<oordinatc positions of 
the correspondiDg prior pixel in pixel blodc 21Qa. 

Function block 236 represents an inquiry as to whether 
object 204^ includes any remaining pixels. Whenever object 
204^ includes remaining pixels, function block 236 pro- 
ceeds to function block 238 and otherwise proceeds to end 
block 240. 

Function block 238 indicates that a next pixel in object 
204^ is identified according to a predetermined format or 
sequence. With the initial pixel selected as described above 
in reference to function block 214, subsequent pixels may be 
defined by first identifying the next adjacent pixel in a row 
(Le., of a common y-coordinate value) and, if object 204 
includes no other pixels in a row, proceeding to the first or 
left-most pixel (i.e., of minimum x-coordinate value) in a 
next lower row. The pixel so identified is designated the 
current pixel and function block 238 returns to fiinctlon 
block 222. 

Polygon block method 200 accurately identifies corre- 
sponding pixels even if they are located at or near an object 
perimeter. A significant source of error in conventional block 
matching jnocesses is eliminated by omitting or disregard- 
ing pixels of pixel blocks 210b falling outside object 204b. 
Conventional block matching processes rigidly apply a 
unifonn pixel block configuration and are not applied with 
reference to a segmented object The unifonn block con- 
figuratiODS cause significant errors for pixels adjacent the 
perimeter of an object because the pixels outside the object 
can undergo significant changes as the object moves or its 
background changes. With such extraneous pixel variations 
induded in conventional block matching processes, pixels in 
the vicinity of an object perimeter cannot be correlated 
accurately with the corresponding pixels in prior image 
fi^mcs. 

For each pixel in object 204b, a corresponding prior pixel 
in object 204a is identified by comparing pixel block 210b 
with a pixel block 210a for each of the pixels in prior object 
204a. The conesponding prior pixel is the pixel in object 
204a having the pixel block 210a that best correlates to pixel 
block 210b. If processed in a conventional manner, such a 
determination can require substantial computation to iden- 
tify each coiresponding pric»r pixel. To illustrate, for pixel 
blocks having dimensions of nxn pixels, which are signifi- 
cantly smaller than a search area 212 having dimensions of 
mxm pixels, approximately n^xm^ calculations are required 
to identify each corresponding prior pixel in the prior object 
204a. 

Pixel Block Correlation IVocess 

FIG. 8 is a functional block diagram of a modified pixel 
block correlation process 260 that preferably is substituted 
for the one described with reference to function block 232. 
Modified correlation process 260 utilizes redundancy Inher- 
ent in correlating pixel blocks 210b and 210a to significantly 
reduce the number of calculations required. 

Correlation process 260 is described with reference to 
FIGS. 9A-9G and lOA-lOG, which schematically rcpre- 
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sent arbitrary groups of pixels ccm'esponding to successive 
image frames 202a and 202b. In particular, FIG. 9 A is a 
schematic representation of a pixel block 262 having dimen- 
sions of 5x5 pixels in which each letter corresponds to a 

5 different pixel. The pixels of pixel block 262 are arranged as 
a right regular array of pixels that includes distinct columns 
264. FIG. 9B represents an anay of pixels 266 having 
dimensions of qxq pixels and corresponding to a search area 
212 in a prior image frame 202a. Each of the numerals in 

10 FIG. 9B represents a different pixeL Although described 
with refa-ence to a conventional right regular pixel block 
262, correlation process 260 is similariy applicable to 
polygonal pixel blocks of the type described with reference 
to polygon match process 200. 

1^ Function block 268 indicates that an initial pixel block 
(e.g., pixel block 262) is defined with respect to a central 
pixel M and scanned across a search area 212 (e.g., pixel 
array 266) generally in a raster pattern (partly shown in FIG. 
7A) as in a conventional block match process. FIGS. 9G-9G 

20 schematically illustrate five of the approximately q^ steps in 
the blodc matdiing process between pixel block 262 and 
pixel array 266. 

Althoug;h the scanning of pixel block 262 across pixel 
array 266 is performed in a conventional manner, oompu- 
tations relating to the correlation between them are per- 
formed differently in this implementation. In particular, a 
correlation (e.g., a mean absolute error) is detccroined and 
stored for each column 264 of pixel block 262 in each scan 
position. The correlation that is determined and stored for 
each column 264 of pixel block 262 in each scanned position 
is referred to as a column correlation 270, several of which 
are symbolically indicated in FIGS. 9C-9G by referring to 
the correlated pixels. To illustrate, FIG. 9C shows a column 
correlation 270(1) that is determiDcd for the single column 
264 of pixel block 262 aligned with pixel array 266. 
Similarly, FIG. 9D shows column correlations 270(2) and 
270(3) that are determined for the two columns 264 of pixel 
block 262 aligned with pixel array 266. FIGS. 9E-9G show 
similar column correlations with pixel block 262 at three 

^ exemplary subsequent scan positions relative to pixel array 
266. 

The scanning of initial pixel block 262 over pixel array 
266 provides a stored array or database of column conela- 
tions. With pixel block 262 having r-number of columns 
264, and pixel array 266 having qxq pixcb, the colunm 
correlation database includes approximately rq^ number of 
column correlations. Hiis number of column correlations is 
only approximate because pixel block 262 preferably is 
initially scanned across pixel array 266 such that pixel M is 
aligned with the first row of pixels in pixel array 266. 

The remaining steps beginning with the one indicated in 
FIG. 9C occur after two conoplctc scans of pixel block 262 
across piicel array 266 (ix., with pixel M aligned with the 

35 first and second rows of pixel array 266). 

Function block 274 indicates that a next pixel block 276 
(FIG. lOA) is defined from, for example, image frame 202b 
witfarespect to a central pixel N in the same row as pixel M. 
Pixel block 276 includes a column 278 of pixels not included 

60 in pixel block 262 and columns 280 of pixels included in 
pixel block 262. Pixel block 276 docs not include a column 
282 (FIG. 9A) that was included in pixel block 262. Such an 
incremental definition of next pixel block 276 is substan- 
tially the same as that used in conventional block matching 

65 processes. 

Function block 284 indicates that pixel block 276 is 
scanned across pixel array 266 in the manner described 
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above with reference to function block 268. As witii FIGS. improvement of this method over conventional calculation 
9C-9G, FIGS. lOB-lOG represent the scanning of pixel methods is even greater. Conventional block matching pro- 
block 276 across pixel array 266. cesses identify only total block correlations for each scan 

Function block 286 indicates that for column 278 a position of initial pixel block 262 relative to pixel array 266. 

column coiTclation is determined and stored at cadi scan S As a consequence, all coirdation values for all pixels must 

position. Accordingly, column coirelations 288(l)-288(5) be calculated separately for each scan position. In contrast 

arc made with respect to the scanned positions of column correlation process 260 utilizes stored column correlations 

278 shown in respective FIGS. lOB-lOF. 270 to significantly reduce the number of calculations 

Functi(Hi block 290 indicates that for each of columns 280 required. The improvements in speed and processor resource 
in pixel block 276 a stored column detcnnination is retrieved requirements provided by correlation process 260 more than 
for each scan position previously confuted and stored in system requirements for staring the column car- 
function block 268. F<ff exan^)le, colunm correlation 270(1) relations. 

of FIG. 9C is the same as column correlation 270'(1) of FIG. It will be appreciated that cocrclation process 260 has 

IOC. Similarly, column correlations 270'(2), 270'(3), 270' been described with reference to FIGS. 9-11 to illustrate 

(5)-270'(8). and 270'(15)-270'(18) of FIGS. lOD-lOF are specific features of this implementation. As shown in the 

the sanne as the con^esponding column correlations in FIGS. illustrations, this iinplementation includes recurring or 

9D, 9E, and 9G. For pixel block 276^ therefore, only one cyclic features that are particularly suited to execution by a 

column correlation 288 is calculated for eadi scan position. computer system. These recurring or cyclic features are 

As a result, the number of calculations required for pixel dependent upon the dimensions of pixd blocks and pixel 

block 276 is reduced by nearly 80 percent ^ arrays and arc well understood and can be implemented by 

Function block 292 indicates that a subsequent pixel persons skilled in the art. 
block 294 (FIG. IIA) is defined with respect to a central 

pixel R in the next successive row relative to pixel M. Pixel Multi-Dimcnsional Transformation 

block 294 includes columns 296 of pixels that are similar to HG. 12 is a functional block diagram of a transformaUon 

but distinct firom columns 264 of pixels in pixel block 262 " method 350 that Includes generating a multi-dimensional 

of HG. 9A. In particular, colunms 296 include pixels A*-E* transformation between objects in first and second succes- 

not included in columns 264. Such an incremental definition give image fi-amcs and quantitizing the mapping for trans- 

of subsequent pixel block 294 is substantially the same as mission or storage. The multi-dimensionai transformation 

that used in conventional block matching processes. ^ preferably Is utilized in connection with function block 96 of 

Function block 298 indicates that pixel block 294 is FIG. 3. Transformation method 350 is described with ref- 

scanned across pixel array 266 (FIG. 9B) in the manner erence to FIG. 7A and FIG. 13, the latter of which like FIG. 

described above with reference to function blocks 268 and 7B is a simplified representation of display screen 50 

276. FIGS. IIB-IIF represent tiic scanning of pixel block showing image firame 202b in which image feature 204 is 

294 across pixel array 266. rendered as object 204b. 

Function block 300 indicates that a column correlation is TVansformation method 350 f^eferably provides a multi- 
determined and stored for each of columns 296. dimensional aflBne transformation capable of representing 
Accordingly, column correlations 302(1)-302(18) are made complex motion that includes any or aU of translation, 
with respect to the scanned positions of columns 296 shown roution, magnification, and shear. Transformation method 
in FIGS. IIB-IIF. ^ 350 provides a significant irnprovemcnt over conventional 

Each of column correlations 302(1)-302(18) may be video compression methods such a MPEG- 1, MPEG-2, and 

calculated in an abbreviated manner with reference to col- H.26X, which are of only one dimension and represent only 

umn correlations made with respect to pixel block 262 (FIG. translation. In this regard, the dimensionality of a transf cr- 

mation refers to the number of coorxlinates in the generalized 

For example, column correlations 302(4)-302(8) of FIG. 43 form of the transformation, as described below in greater 

IID include subcolumn cosxelations 304'(4)-304'(8) that are detail. Increasing the aocuiacy with which complex motion 

the same as subcolumn conelations 304(4)-304(8) of FIG. is represented results in fewer errors than by conventional 

9E. Accordingly, column correlations 302(4)-302(8) may be representations, thereby increasing compression efficiency, 

dctmnined fromrespective column conelations 270(4)-270 Rmction block 352 indicates that a dense motion estima- 

(8) by subtracting from tfie latter correlation values for 50 tion of the pixels in objects 204fl and 204b is determined, 

pixels OlA, 02B, 03C» 04D, and 05E to form subcolumn Preferably, the dense motion estimation is obtained by 

correlations 304(4>-.304(8), respectively. Column correla. polygon match process 200. As described above, flie dense 

tions 302(4)-302(8) may be obtained by adding correlation motion estimation includes motion vectors between pixels at 

values for the pixel pairs 56A', 57B',58C, 59D' and SOE to cocrdinates (x,, y^,) in object 204b of image frame 202b and 

the respective subcolumn correlation values 304(4)-304(8), 55 corresponding pixels at locations y/) of object 204a in 

respectively. image frame 202fl. 

The determination of colurmi correlations 302(4)-302(8) Function block 354 indicates that an array of transforma- 

from respective column correlations 270(4>-270(8) entails tion blocks 356 is defined to enconmass object 204b 

subtracting individual pixel correlation values correspond- preferably, transformation blocks 356 are right regular 

ing to the row of pixels A-B of pixel block 262 not included 60 arrays of pixels having dimensions of, for example, 32x32 

in pixel block 294, and adding pixel cccrelation values for pixels. 

the row of pixels A-E* Included in pixel block 25M but not i7„ ut^^y, «o • ^ * * 

pixelblock262.Thismcthodsubsti£tesforea<*ofcolumn mdicales ti^ a multi-dmiensional 

correlations 302(4).302(8), one substraction and one addl- ^ ^^ZV^^'^ 't . ca* transfom^on 

tion for the five additions that would be required to deter- 65 f ' transformations are of first 
mine each colunm correlation in a conventional manneL 

With pixel blocks of larger dimensions as are preferred, the xt'=ea^y:H: 
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and arc dctcnnincd with reference to all pixcb for which the 
motion vectors have a relatively high confidence. These 
afiSne transfcrmations are of two dimensions in that and 

arc defined relative to two coordinates: and y^. 

The relative confideace of the motion vectors refers to the 
accuracy with which the motion vector between conespond- 
ing pixels can be dctcnnincd uniquely relative to other 
pixels. For example, motion vectors between particular 
pxels that are in relatively large pixel anays and are 
uniformly colored (e.g., black) cannot typically be deter- 
mined accurately. In particular, for a black pixel in a first 
image frame, many pixels in the pixel array of the subse- 
quent image frame will have the same correlation (i.e.. mean 
absolute value error between pixel blocks). 

In contrast, pixel arrays in which pixels correspond to 
distinguishing features typically will have relatively high 
coirelations for particular corresponding pixels in succes- 
sive image frames. 

The relatively high correlations are preferably represented 
as a minimal absolute value error determination for particu- 
lar pixel. Motion vectors of relatively high confidence may, 
therefore, be deteimined relative to such uniquely low error 
values. For example, a high confidence motion vector may 
be defined as one in which the minimum absolute value error 
for the motion vector is less dian the next greater ecror value 
associated widi the pixel by a difference amount that is 
greater than a threshold difference amount Alternatively, 
high confidence motion vectors may be defined widi respect 
to the second order derivative of the absolute eaor values 
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Preferably these equations are solved by a conventional 
singular value decomposition (SVD) method, which pro- 
vides a minimal least-square error for the ^proximation of 
the dense motion vectors. A conventional SVD method is 
described, for example, in Numerical Recipes in C, by Ptess 
et aL, Cambridge University Press, (1992). 

As described above, the preferred two-dimensional affine 
transformation equations are capable of representing 
translation, rotation, magnification, and shear of transfor- 
mation blocks 356 between successive image frames 202a 
and 202^. In contrast, conventional motion transformation 
methods used in prior compression standards employ sim- 
plified transformation equations of the form: 

The prior simplified transformation equations represent 
motion by only two coefficients, g and h, which represents 
only one-third the amount of information (Le., coefficients) 
obtained by the preferred multi-dimensional transformation 
equations. To obtain superior compression of the informa- 
tion obtained by transformation method 350 relative to 



upoD which the correlations are (ietenBined. A second order ~nventionalcoii«^sdonii>«^ 

formation block 356 preferably are more than three tmies 



derivative of more than a particular value would indicate a 
relatively high correlation between specific corresponding 
pixels. 



larger than the cOTesponding 16x16 pixel blocks employed 
in MPEG-1 and MFEG-2 con^iression methods. TTie pre- 
ferred 32x32 pixel dimensions of transfonnation blocks 356 



With n-numbcr of pixels with such high-confidence ^ encompass four times the number of pixels employed in the 



motion vectors, the preferred affine transformation equations 
are solved with reference to n-number of corresponding 
pixels in image frames 202a and 2102b. foages frames must 
include at least ^trte corresponding pixels in image frames 



transformation blocks of conventional transformation meth- 
ods. The larger dimensions of transformation blocks 356, 
together with the improved accuracy with which the afiSne 
transformation coefficients represent motion of the transfor- 



202aand2026withhighconftdenccmotionvectarsto solve « '^?" ^56, aUow transfonnatioii method 3S0 to 

provide greater con^n'ession than conventional compression 
methods. 



for the six unknown ooefficicnts a, b, c, d, c, and f of the 
preferred affme transformation equations. With the preferred 
dimensions, each of transf(Hmation blocks 356 includes 2^° 
pixels of which significant nimibers typically have relatively 
high confidence motion vectors. Accordingly, the affine 
transformation equations are over-determined in that a sig- 
nificantly greater number of pixels are available to solve for 
the coefficients a. b, c, d, e, and f. 

The resulting n-numtser of equations may be rq>resented 
by the linear algebraic expression: 
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It will be appreciated that the affine coefficients generated 
typically would be non-integer, floating point values that 
could be difficult to confess adequately without adversely 
affecting their accuracy. Accordingly, It is preferable to 
quantize the affine transformation coefficient to reduce the 
bandwidth required to store or transmit them. 

Function block 362 indicates that the affine transforma- 
tion coefficients generated with reference to function block 
358 are quantized to reduce the bandwidth required to store 
or transmit them. FIG. 14 is an enlarged fragmentary rep- 
resentation of a transformation block 356 showing three 
selected pixels, 364a, 3646, and 364c from which ttie six 
pref ened affine transformation coefficients a-f may be deter- 
mined. 

Pixels 364a-364c are represented as pixel coordinates 
i^u yO* (^2> (^3» yy)y respectively. Based ^)on the 

dense motion estimation of function block 352, pixels 
65 364a-364c have respective corresponding pixels (x^*, y,'), 
(y^j ya')) (^3^3) in preceding image frame 202a . As is 
conventional, pixel locations (x,-, y^) are represented by 



60 
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integer values and are solutions to ttie affine transfonnation 
equations upoo which the j^efored afOne transfomiatioD 
coefficients are based. Accordingly, selected pixels 
364a-364c are used to calculate the ccrrespoading pixels 
from the preceding image frame 202a, which typically will 3 
be floating point values. 

Quantization of these floating point values is performed 
by converting to integer format the difference between 
corresponding pixels (x,-x*i, y.-y't). The afBnc transforma- 
tion coefficients are determined by first calculating &e pixel lO 
values (x*,. y'f) from the difference vectors and the pixel 
values (Xj, y^), and then solving the multi-dimensional 
transformation equations of function block 558 with respect 
to the pixel values (x',-, y*,). 

As shown in FIG. 14, pixels 364a-364c preferably are 15 
distributed about transformatioo block 356 to minimize tiic 
sensitivity of the quantization to local variations within 
transfonnation block 356, ftcfcrably, pixel 364a is posi- 
tioned at or adjacent the center of transformation block 356, 
and pixels 364t and 364c are positioned at upper comers. 20 
Also in the preferred embodiment, the selected pixels fco* 
cadi of the transfcrmatioD blocks 356 in object 204& have 
the same positions, thereby allowing the quantization pro- 
cess to be performed efficiently. 

Another aspect of the quantization method of function 25 
block 362 is that different levels of quantization may be used 
to represent varying degrees of motion. As a result, rela- 
tively simple motion (e.g., translation) may be represented 
by fewer selected pixels 364 than are required to represent 
coiiq)lex motion. With respect to the affine transformation 30 
equations described above, pixel 364fl (x,-, y^ from object 
204fe and the corresponding pixel (x^*, y j') from object 204a 
are sufficient to solve simplified affine transfonnation equa- 
tions of the form: 



which represent translation between successive image 40 
frames. Pixel 364a spccificaUy is used because its central 
position generally represents translational motion indepen- 
dent of the other types of motioa Accordingly, a user may 
selectively represent simplified motion such as translation 
with simplified affine transformation equations that require 4S 
one-third the data required to represent complex motion. 

Similarly, a pair of selected pixels (Xj, y^) (e.g., pixel 
364a ) and (Xj, (i.e., either of pixels 364^ and 364c) from 
object 2046 and the corresponding pixels (x/, y^') and (Xj , 
y}') from object 204a are sufficient to solve sinq)lified affine 50 
transformatioD equations of the form: 

which are capable of representing motions that include 
translation and magnification between successive image 
frames. In the simplified form: 

60 

y^sm OxHi cos 9yi/ 

the corresponding pairs of selected pixels are capable of 63 
representing motions that include translation, rotation, and 
isotropic magnification. In this sin^lified fonn, the common 



789 

22 

coefficients of the x and y variables allow the equations to 
be solved by two corresponding pairs of pixels. 

Accordingly, a user may selectively represent moderately 
complex motion that includes translation, rotation, and mag- 
nification with partly simplified affine transformation equa- 
tions. Such equations would require two-thirds the data 
required to represent complex motion. Adding ttie third 
selected pixel (X3, y^) from object 204^, the corresponding 
pixel (X3*, y,*) from object 204a, and the complete preferred 
affine transformation equations allows a user also to jcprc- 
sent shear between successive image frames. 

A preferred embodiment of transformation method 350 
(FIG. 12) is described as using uniform transformation 
blocks 356 having dimensions of, for cxany)Ie, 32X32 
pixels. The preferred multi-dimensional affine transforma- 
tions described with reference to function block 358 are 
determined with reference to transformation blocks 356. It 
will be af^iredated that the dimensions of transfcrmation 
blocks 356 directly aflfcct the con^rcssion ratio provided by 
this method. 

Fewer transformation blocks 356 of relatively large 
dimensions are requfred to represent transformations of an 
object between image frames than the number of transfor- 
mation Mocks 356 having smaller dimensions. A conse- 
quence of uniformly large transfcomation blocks 356 is that 
correspondingly greater error can be introduced for each 
transformation block. Accordingly, uniformly sized trans- 
formation blocks 356 typically have moderate dimensions to 
balance these confficting performance constraints. 

lYansformation Block Optimization 

FIG. 15 is a functional block diagram of a transformation 
block optimization method 370 that automatically selects 
transfonnation block dimensions that provide a minimal 
error threshold. Optimization method 370 is described with 
reference to FIG. 16, which is a sin^jlified representation of 
display saeen 50 showing a portion of image frame TJdlb 
with object 204£>. 

Function block 372 indicates that an initial transformation 
block 374 is defined with respect to object 204b. Initial 
transfonnation block 374 preferably is of maximal dimen- 
sions that are sdectable by a user and are, for example, 
64x64 pixels. Initial transformation block 374 is designated 
the cuuent transformation block. 

Function block 376 indicates that a current signal-to-nolsc 
ratio (CSNR) is calculated with respect to the current 
transformation block. Hie signal-to-noise ratio preferably is 
calculated as the ratio of the variance of the color component 
values of the pixel within the current transformation block 
(i.e., the signal) to the variance of the color components 
values of the pixels associated with estimated error 98 (FIG. 
3). 

Function block 378 indicates that tiie current transforma- 
tion block (e.g., transformation block 374) Is subdivided 
into, for exasaplt^ four equal sub-blocks 380a-380J, affine 
transformations are determined for each of sub-blocks 
380a-380^, and a future fiignal-to-noise ratio is determined 
with respect to the affine transformatioDS. The future signal- 
to-noise ratio is calculated in substantially the same manner 
as the current signal-to-noise ratio described with reference 
to function block 376. 

Inquiry block 382 represents an inquiry as to whether the 
future signal-to-noise ratio is greater than the current signal- 
to-noise ratio by more than a user-selected threshold 
amount. T\ds inquiry represents a determination that further 
subdivision of the current transformation block (e.g., trans- 
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fonnadon block: 374) would improve the accuracy of the 
afi&ne transfonnations by at least the threshold amount 
Whenever the future signal-to-noise ratio is greater than the 
cuzrent signal-to-noise ratio by more than the threshold 
amount, inquiry blodc 382 proceeds to function block 384, 
and otherwise proceeds to function block 388. 

Function block 384 indicates that sub-blocks 380a-380J 
are successively designated the current transfonnation 
block, and each are analyzed whetba to be further subdi- 
vided For purposes of iUustration, sub-block 380a is des- 
ignated the cuirent transformation and processed according 
to function block 376 and further sulxlivided into sub- 
blocks 3860-386^. Function block 388 indicates that a next 
successive transfonnation block 374' is identified and des- 
ignated an initial or current transfonnation block. 

Prccomprcssion Extracompression E^polation 
Method 

FIGS. 17A and B arc a functional block diagram of a 
precompression extri^lation method 400 for extr^x>lating 
image features of axbitrary configuration to a predefined 
configuration to fadiitate concession in accordance with 
function block 112 of encoder process 64 (both of FIG. 3). 
Extrapolation method 400 allows the compression of func- 
tion block 112 to be performed in a conventional manner 
such as DCT or lattice wavelet compression, as described 
above. 

Conventional still image compression methods such a 
lattice wavela compression or discrete cosine transforms 
(DCT) operate upon rectangular arrays of pixels. The meth- 
ods described here are applicable to image features or 
objects of arbitrary configuration. Extrapolating such objects 
or image features to a rectangular pixel array configuration 
allows use of conventional still image concession methods 
such as lattice wavelet compression or DCT. Extr^olation 
method 400 is described below with reference to FIGS. 
18A-18D, which are rqaresenutions of display screen 50 on 
which a simple object 402 is rendered to show various 
aspects of extrapolation method 400. 

Function block 404 indicates that an extrapolation block 
boundary 406 is defined about object 402. Extrapolation 
block boundary 406 preferably is rectangular. Referring to 
RG. 18A, the formation of extrapolation block boundary 
406 about object 402 is based upon an identification of a 
perimeter 408 cf object 402 by, for example, object seg- 
mentation method 140 (FIG. 4). Extr^lation block bound- 
ary 406 is shown encompassing object 402 in its entirety for 
purposes of illustration. It will be aic^ciated that extrapo- 
lation block boundary 406 could alternatively encompass 
only a portion of object 402. As described with reference to 
object segmentation method 140, pixels included in object 
402 have color component values that differ from those of 
pixels not included in object 402. 

Function block 410 indicates that all pixels 412 bounded 
by extrapolation block boundary 406 and not included in 
object 402 are assigned a predefined value such as, for 
example, a zero value for each of the color oonq>oi^cnts. 

Function block 414 indicates that horizontal lines of 
pixels within extrapolation block boundary 406 are scanned 
to identify horizontal lines with horizontal pixel segments 
having both zero and non-zero color con^^nent values. 

Function block 416 represents an inquiry as to whether 
the horizontal piKcl segments haying color con^nent val- 
ues of zero arc bounded at both ends by perimeter 408 erf 
object 402. Referring to FIG. 18B, region 418 represents 
horizontal pixel segments having coles' coiiq>onent values of 
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uao that arc bounded at both ends by pcrimetca' 408. 
Regions 420 represent horizontal pixel segments that have 
color component values of zero and are bounded at only one 
end by perimeter 408. Fiinction block 416 proceeds to 

2 function block 426 for regions 418 in whidi the pixel 
segments have color component values of zero bounded at 
both ends by perimeter 408 of object 402, and otherwise 
proceeds to function block 42Z 
Function block 422 indicates that the pixels in each 

10 horizontal pixel segment of a region 420 is assigned the 
color oonq>onent values of a pixel 424 (only exemplary ones 
shown) in the corresponding hcsizontal lines and perimeter 
408 of object 402. Alternatively, the color conqwnent values 
assigned to the pixels in regions 420 are functionally related 

1^ to the color component values of pixels 424. 

Function block 426 indicates that the pixels in each 
horizontal pixel segment in region 418 are assigned color 
component values corresponding to, and preferably equal to, 
an average of the color component values of pixels 428a and 

^ 428^ that are in the corresponding horizontal lines and on 
perimeter 408. 

Function block 430 indicates that vertical lines of pixels 
within extrapolatiOD block boundary 406 are scanned to 
identify vertical lines with vertical pixel segments having 

^ both zero and non-zero color component values. 

Function block 432 represents an inquiry as to whether 
the vertical pixel segments in vertical lines having color 
component values of zero arc bounded at both ends by 

^ perimeter 408 of object 402. Referring to FIG. 18C, region 
434 represents vertical pixel segments having color compo- 
nent values of zero that are boimded at both ends by 
perimeter 408. Regions 436 represent vertical pixel seg- 
ments that have color component values of zero and are 
bounded at only one end by perimeter 408. Flmction block 
432 proceeds to function block 444 for region 434 in which 
the vertical pixel segments have color component values of 
zero bounded at both ends by perimeter 408 of object 402, 
and otfamvise proceeds to function block 438. 

^ Function block 438 indicates that the pixels in each 
vertical pixel segment of region 436 are assigned the color 
component values of pixels 442 (only exemplary ones 
shown) in the vertical lines and perimeter 408 of object 46z 
Alternatively, the color component values assigned to the 
pixels in region 436 are functionally related to the color 
component values of pixels 442. 

Function block 444 indicates that the pixds in each 
vertical pixel segment in region 434 are assigned color 
conq>onent values corresponding to, and preferably equal to, 

3g an average of the color component values of pixels 446a and 
446fr that are in the harizontal lines and on perimeter 408. 

Function block 448 indicates that pixels that are in both 
horizontal and vertical pixel segments that are assigned 
color conoponent values according to this method are 

55 assigned conq>osite color component values that relate to» 
and preferably are the average of, the color component 
values otherwise assigned to the pixels according to their 
horizontal and vertical pixel segments. 

Examples of pixels assigned such composite color com- 

60 ponent values are those pixels in regions 418 and 434. 
Function block 450 indicates that regions 452 of pixels 
bounded by extrapolation block boundary 406 and not 
intersecting perimeter 408 of object 402 along a horizontal 
or vertical line are assigned con^site color component 

65 values that are related to, and preferably equal to the average 
of, the color conq)onent values assigned to adjacent pixels. 
Referring to FIG. 18D, each of pixels 454 in regions 452 is 
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assigned a color component value ttiat preferably is the additional compression. FIG. 20C represents a resulting 

average of the color component values of pixds 456a and compressed set of pixels 515. 

4566 that are aligned with pixel 454 along respective A 2x2 up sample filter 516 inserts a pixel of zero value in 

horizontal and vertical lines and have non-zeio color com- place of each pixel 512 omitted by down sampling filter 514« 

ponent values previously assigned by this method. s and intexpolation filter 518 assigns to the zero-value pixel a 

A benefit of object extrapolation process 400 is that is P^^^ average of the opposed adjacent pixels, or 

assigns smoothly varying color con^xment values to pixels f Fevious assigned value if the zcro^valuc pixel is not 

not included in object 402 and therefore optimizes the !?f^*^'^° ^i^?^*^ ""^^'^T^ P"^*^'* ^% 

compression capabilities and accuracy of conventional still ^e "b^ek 520 

image compression methods. In contrast, prior ait zoo lo . ^ ' . , ^ ^ , 

padding or minor image methods, as described by Chang et ^ difference 522 is taken between the color component 

al„ *Transform Coding of Arbitrarily-Shapcd Image f ^ ^^P^^^^ jjj^ comisponding color 

Segments," ACM Multimedia, pp. 83-88, June, 1993. apply ^o^opontLt values for set of pixels 520 to fonn a zero-order 

compression to extrapolated objects that are filled with componcat 

pixels having zero color con^wnents values such as those is A second decimation filter 526 receives color component 

^lied in function block 410. The drastic image change corresponding to the compressed set of pixels 515 

than occurs between an object and the zero-padded padded generated by first 2x2 down sampling filter 514. Decimation 

regions introduces high frequency changes that are difficult preferably is the same as decimation filter 502 

to compress or introduce image artifacts upon compression. (^-S ' * nonseparable median filter). Accordingly, deci- 

Object extrapolation method 400 overcomes such disadvan- 20 mation filter 526 functions in the same manner as decima- 

X^g^^ tion filter 502 and delivers a resulting compressed set or 

array of pixels (not shown) to a second 2x2 down san^ling 

Alternative Encoder Method filter 528. 

no. 19A is a ftmctional block diagram of an encoder Down sampling filter 528 functions in the same manner as 

method 500 that employs a Laplacian pyramid encoder with ^ ^^"^^ sampling filter 514 and forms a second order image 

unique filters that maintain nonlinear aspects of image component that also is delivered to a 2x2 up sample filter 

features, such as edges, while also providing high compres- ^ interpolation filter 531 that function in the same 

sion. Conventional Laplacian pyramid encoders arc inanna as up sample fiUer 516 and inter^^ 

described, for exanmle, in the Laplacian Pyramid as a respectively. A difference 532 is taken between the color 

Compact Image Code by Burt and Addleson, IEEE TYans. 3° component values of the set of pixels 515 and the resulting 

Comm., Vol. 31, No. 4, pp. 532-540, April 1983. Encoder color con^onent values provided by interpolation filter 531 

method 500 is capable of providing the encoding described ^ ^^"^ » first-order image component 1^. 

with reference to function block 112 of video compression Th^ "^g« components Iq. Ii, and Lj arc respective 
encoder process 64 shown in FIG. 3, as well as whenever 

else DCr on wavelet encoding is suggested or used. By way -2- x-^ » -J '-J- 
of example, encoder method 500 is described with reference 
to encoding of estimated eiror 110 (FIG. 3). 

A first decimation filter 502 receives p«el infonnation ''^ con^nent values that represent the color 

coiresponding to an estimated error 110 (HO. 3) and filters component values for an nxn array of pixek 504. 

the pixels according to a filter criterion. In a conventional « ^^S' component mamtains the high fequency coin- 

Lapladan pyramid method, Ac decimation filter is a low- Pon«ts (e g edges) of an image representel by the origtaal 

pass filter in* as a Gaussian weighting function. In accor- ^04. Image components I, and represent low 

dance with encoder method 500, however, dedmation fllt« fr««!"en«y aspects of the oiigmal miage. Image components 

502 preferably employs a median filter and, more ?o. Ii and provide relative compression of flie ongmal 

specifically, a 3x3 nonseparable median filter. "^Ec Image coinponent !„ and I^ maintain high frequency 

_ .„ . . - . . . . . ^ ^. features (e.g., edges) in a foimat that IS highly compressible 

Toinustratc,nG. 20Aisasmipkfiedrcpresent^^^^ ^ the relatively high correlation betsJc^ the v^ues of 

colorcon^nentvaluesforonecolorcoiiv^^^ adjacent pixek. Image^mponent L, is not readfly oom- 

for an arbitrary set or array of pixels 504. Although pjcssible because it IndudeTpriinarily low frequency im^^ 

described witii p^cular reference to red color component ^ ^ ^ ^^J^^^ 

values, this dlustraUon is sumljly apphcd to the green and ^ ^ funcdonBl block diagram of a decoder 

blue color component values of pixels 504. ^^^^ ^ ^^^^ ^^^^ 

With reference to the preferred embodiment of dcdma- ^ents Iq, I^, and L^ generated by encoder method 500. 

tion filter 502, filter blocks 506 having dimensions of 3x3 Decoder m^od 53li includes a first 2x2 up sample filter 538 

pixels are defined among pixels 504. For each pixel block 55 ^ receives image component Lj and interposes a pixel of 

506, the median pixel intensity value is identified or ^cro value between each adjacent pair of pixels. An intcr- 

selectcd. With reference to pixel blocks 506fl-506c, for polation filter 539 assigns to the zero-value pixel a pixel 

example, decimation filter 502 provides Ac respective val- yg^^ ^ preferably is an average of the values cf the 

ucs of 8» 9, and 10, which are Usted as the first three pixels adjacent pucls, or a previous assigned value If the zero- 

512 in FIG. 20B. ^ value pixel is not between an opposed pair of non-zero- value 

It will be appreciated, however, that decimation filter 502 pixels. First 2x2 up sample filter 538 operates in subslan- 

could employ other median fillers. Accordingly, for each Ually the same manner as up sample filters 516 and 530 of 

group of pixels having associated color component values of FIG. 19A, and interpolation filter 539 operates in substan- 

{ao, a^, . . . , a^J the median filler would select a median tially the same manner as interpolation filters 518 and 531. 

value aji^ 65 A sum 540 is determined between image component 

A first 2x2 down sampling filter 514 samples alternate and the color component values corresponding to the 

pixels 512 in vertical and horizontal directions to provide deconqjressed set of pixels generated by first 2x2 up sample 
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iiltcr 538 and intapolatioD filter 559. A secoad 2x2 up conventioaal extrapolation methods, such as a mirror image 

sample filter 542 inteiposes a pixel of zero value between method, could alternatively be utilized, 

each adjacent pair of pixels generated by sum 540. An Function block 566 indicates that the dense motion vector 

interpolation filter 543 assigns to the zero-value pixel a pixel ft^d with its extrapolated regular configuration is encoded 

value that includes an average of the values of tiie adjacent 5 or compressed according to conventional encoding transfor- 

pixcb, or a previous assigned value if the zero-value pixel mations such as, for exanq)le, discrete cosine transformation 

is not between an opposed pair of non-zo^o-value pixels. Up (per) cr lattice wavelet con^>ression, the former of which 

sample filter 542 and interpolation filter 543 are substan- is prefexred 

S^^^Sr"'''""*'^''*''"^^*'"''"'"^'^"'''"'" Rmction Mock 568 indicates that the eiico<»ed dense 

.resprenv y- . ,« ^ , '° motion vector field is ftifter cwnmresscd or encoded by a 

A sura 544 sums the unage ^inponeDt 10 with the color conventional lossless stiU image cTnifression method such 

component values coirespon<hng to the decoi,y«ssed set of ^ ^ an ei^Scnse motion vector 

p«eb generated by second 2)^up san^e filter 542 ^ sucJ, I stiU image compression method is 

mteipoUtion filter 543. Sum 544 provides decompressed ^^^^ ^^^^^ ^ functionXk U4 of FIG. 3. 

estimated error 110 corresponding to the estmiated error 110 15 ^ ^ 

deUvercd to encoder process 500. Con^ssion of Quantized Objects B-om Previous 

TVansfonn Coding of Motion Vectors Vidoc Frames 

^ , . . . . Referring to FIG. 3, video cortmression encoder process 

Conventional v^o^mp^ession encoder pn>cesses, such ^ ^^.^^ 98 determined with reference 

as MPEG-1 or MEEG-2. utihze only sparse motion vector ^ ^ , coixesponding object in a 

fields to represent the motion of signiftonUy 1^« p^el „ext successive frame N. As a consequent enc^process 

arrays of a regular Size and configuration. The motion vector ^l* ^ ^ ^ 

^ , / zL • * 1 *f » I J . ^ requires that quantized pnor object 98 be stored in an 

folds are q.arse u. th^ only one motioD vector is used to ^.^^^y, „^ ^„ conventional video display 

represent toe mohwi of a p«el army ha«ng dmiensions of, resolutions, such a memory buffer would require a cap^Q^ 

for«^e 1^16 pads, m sparse ^^^^^ of atleast one megabyte to store the quanteed pH«^bjert 

togetea: with transform encodmg of und«lywg unages« ^ ^ single Wd4 frwne. Higte resolution display 

pixels by. for example, disaete cosine transfocro (DCT) „,~!t,^^,, ^ , ui^iajr 

encoding, provideTnventional video compression eocodl ^j™^ wouldrequlre coffespondingly largermemory buff- 
ing. 

T ^ . J J 30 FIG. 22 is a functional block diagram of a quantized 

In contrast, video compression encoding process 64 (FIG. a ^a^^^a \ ^aa^ . 

3) utilizes dense mo3vector fields in which motion obm^^6«^co<UT(codtc) proems 

vectors are determined for aU. or vinually all. pixds of an '^t'^^T ^^ decompresses quantized pnor objects 98 to 

object. Such dense motion vector fidds s^nificantly "'P'^^ °' " 'l"^"' '^'^ 
improve the accuracy with which motion between coctc- 

sponding pixels is represented. Although the increased accu- Faction block 602 indicates that each quantized object 
racy can significantly reduce the cirars associated with ^ ^ encoded on a block-by-block 
conventional sparse motion vectcr field representations, the manner by a lossy encoding or compression method such as 
additional information included in dense motion vector discrete cosine transform (DCT) encoding or lattice sub- 
fields represent an increase in the amount of information (wavelet) compression. 

representing a video sequence. Therefore, dense motion Function block 604 indicates that the encoded or corn- 
vector fields are themselves con^iressed or encoded to pressed quantized objects are stored in a memory buffer (not 
improve the concession ratio. shown). 

FIG. 21 is a fii actional block diagram of a motion vector Function block 606 indicates that encoded quantized 

encoding process 560 for encoding or compressing motion ^5 objects are retrieved firom the memory buffer in anticipation 

vector fields and, preferably, dense motion vector fields such of processing a corresponding object in a next successive 

as those generated in accordance with dense motion trans- video frame. 

formation 96 of FIG. 3. It will be appreciated that such dense Function block 60S indicates that the encoded quantized 

motion vector fidds frcwn a selected object typically will object is inverse encoded by, for example, EKTT or wavelet 

have greato- continuity or ^'smoothness** than the underlying y, decoding according to the encoding processes emjAoyed 

pixels comsponding to the object As a result, compression with respect to function block 602. 

or encoding of die dense motion vector fields will attain a Codcc process 600 allows the capacity of the concspond- 

greaterconqircssion ratio than would compression wencod- ing memory buffer to be reduced by up to about 80%. 

ing of the undfflying pixels. Moreover, it wiU be appreciated that codec process 600 

Function block 562 indicates that a dense motion vector 55 would be similarly applicable to the decoder process cone- 

field is obtained for an object or a portion of an object in sponding to video compression encoder process 64. 
accordance widi, for example, the processes of function 

block 96 described with reference to FIG. 3. Accordingly, Video Conqircssion Decoder Process Overview 
the dense motion vector field will conespond to an object or Video compression encoder process 64 of FIG. 3 provides 
other image portion of arbitrary configuration or size. ^ encoded or con^iressed representations of video signals 
Function block 564 indicates that the configuration of the corresponding to video sequences of multiple image frames, 
dense motion vector field is cxtrq>olated to a regular, 'Hie compressed representations include object masks 66, 
preferably rectangular, configuration to facilitate encoding feature points 68, affine transform coefficients 104, and 
or conqrcssion. Preferably, the dense motion vector field con^ssed error data 116 from encoder process 64 and 
configuration is extrqx>lated to a regular configuration by 65 compressed master objects 136 from encoder process 130. 
precoiiq>rcssion extrapolation method 400 described with These compressedrepresentations facilitate storage or trans- 
reference to FIGS. 17A and 17B. It will be appreciated that mission of video information, and are capable of achieving 
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compression ratios of up to 300 percent greater than those Function block 728 indicates that the entropy decoded 

achievable by conventional video compression methods error data from function block 726 is further decompressed 

such as MPEG-2. or decoded by a conventional lossy still image compression 

It will be appreciated, however, that retrieving such method conesponding to that utilized in function block 112 

compressed video information from data storage or rccdv- 5 (PIG- In the preferred embodiment, the decompression 

ing transmission of the video information requires that it be ^ decoding of function block 728 is by a lattice subband 

decoded or decompressed to reconstruct the original video (wavelet) process or a disaete cosine transfam (DCT) 

signal so that it can be rendered by a display device such as F^^^s. 

video dispUy device 52 (FIGS. 2A and 2B). As with J^T"" .^"^ 722 provides quantized object 730 for 

conventional encoding processes video information is sulv lo ^ N as the sum of predicted object 720 and quantized 

vi^ signa^ IS encoded or comi^essed reconstraction of the object in subsequent frames. 

^^"^ '^^'^ Function block 734 indicates that quantized object 732 is 

compression decoder pm^^^ assembled with other objects of a cu^ent image frame N to 

infonnation gencratt^ by video coinprcssion encoder prcv ^ decompressed video signal 

cess 64 of FIG. 3. For purposes of consistency with the ^ 

description of encoder process 64, decoder process 700 is Simplified Chanin Encoding 

described with reference to FIGS. 2A and 2B. Decoder Masks, objects, sprites, and otiier graphical features, 

process 700 retrieves from memory or receives as a trans- commonly are represented by their contours. As shown in 

mission encoded video information that includes object ^ and explained with reference to FIG. 5A, for example, 

masks 66, feature points 68, compressed master objects 136, rectangular solid object 56a is bounded by an object pcrim- 

affine transform coefficients 104, and compressed error data eter or contour 142. A conventional process or encoding or 

compressing contours is referred to as chain encoding. 

Decoder process 700 performs operations that are the FIG. 24A shows a conventional eight-point chain code 

inverse of those of encoder process 64 (FIG. 3). 800 from which contours on a conventional recta-linear 

Accordingly, each of the above-described preferred opera- pixel array are defined. Based upon a current pixel location 

tions of encoder process 64 having a decoding counterpart X, a next successive pixel location in the contour extends in 

would similarly be inversed. one of directions 802c-«02fc. The chain code value for the 

Function block 702 indicates that masks 66, feature points next successive pixel is the numeric value corresponding to 

68, transform coefficients 104, and compressed error data the particular direction 802. As examines, the right, hori- 

116 are retrieved from memory or received as a transmission zontal direction 802a corresponds to the chain code value O, 

for processing by decoder process 700. and the downward, vertical direction 802^ corresponds to 

FIG. 23B is a functional block diagram of a master object cbsda code value 6. Any continuous contour can be 

decoder process 704 for decoding or decompressing com- 33 described from eight-point chain code 800. 

pressed master object 136. Function blodc 706 indicates that With refcrwice to FIG. 24B, a contour 804 represented by 

compressed master object data 136 are entropy decoded by pixels 806 designated X and A-G can be encoded in a 

the inverse of the conventional lossless entropy encoding conventional manner by the chain code sequence 

method in function block 134 of FIG. 3B. Function block {00764432}, In particular, beginning from pixel X, pixels A 

708 indicates that the entropy decoded master object from ^ and B are positioned in direction 0 relative to respective 

function block 706 is decoded according to an inverse of the pixels X and A. Pixel C is positioned in direction 7 relative 

conventional lossy wavdet encoding process used in func- to pixel B. Remaining pixels D-G are similariy positioned 

tion block 132 of FIG. 3B. in directions ccnesponding to the chain code values Hsted 

Function block 712 indicates that dense morion above. In a binary r^escntation, each conventional chain 

transformations, preferably multi-dimensional af&ne 45 code value is represented by three digital bits, 

transfonnations, are generated from affiuie coefficients 104. FIG. 25A is a functional block diagram of a chain code 

Breferably, affine coefficients 104 are quantized in accor- process 810 capable ofproviding contour compression ratios 

dance with transfonnation method 350 (FIG. 12), and the at least about twice those of conventional chain code pro- 

affine transformations are generated from the quantized cesses. Chain code process 810 achieves such improved 

affine coefficients by performing the inverse of the opera- 50 compression ratios by limiting the number of chain codes 

tions described with reference to function block 362 (FIG. and defining them relative to the alignment of adjacent pairs 

12). of pixels. Based upon experimentation, it has been dlscov- 

Function block 714 indicates that a quantized form of an cred that the limited chain codes of chain code process 810 

object 716 in a prior frame N-1 (e.g., rectangular solid object directly represent more than 99.8% of pixel alignments of 

S6a in image frame 54a) provided via a timing delay 718 is 55 ^^j^ or mask contours. Special case chain code modifica- 

transfcnned by the dense motion transfonnation to provide tions accommodate the remaining less than 0.2% of pixel 

a predicted form of the object 720 in a current frame N (e.g., aligmnent as described below in greater detail, 

rectangular solid object 56^ in image frame S4b). Function block 816 indicates that a contour is obtained for 

Function block 722 indicates that for image frame N, a mask, object, or sprite. The contour may be obtained, for 

predicted current object 720 is added to a quantized error 60 example, by object segmentation process 140 described with 

724 generated from con^ressed error data 116. In particular, reference to FIGS. 4 and 5. 

function block 726 indicates that compressed error data 116 Function block 818 indicates that an initial pixel in the 

is decoded by an inverse process to that of compression contour is identified. The initial pixel may be identified by 

process 114 (FIG. 3A). In the prefeired embodiment, func- common methods such as, for example, a pixel with minimal 

tioD blocks 114 and 726 arc based upon a conventional 63 X-axis and Y-axis coordinate positions, 

lossless still image comjffession method such as entropy Function block 820 indicates that a predetermined diain 

encoding. code is assigned to represent the relationship between the 
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initial pixd and the next adjacent pixel in the cx)ntour. code. Whenever there is another pixel in the contour to be 

lYeferabiy, the predetennined chain code ccsresponds to a assigned a chain code, function block returns to function 

forward direction, block 836, and otherwise proceeds to function block 862. 

FIG. 25B is a diagrammatic rqxrcscntation of a three^ Function block 862 indicates that nonoonformal pixel 

point chain code 822. Chain code 822 includes dircc chain 5 alignment directions introduced or incurred by the process 

codes 824a. 824fc, and 824c that coirespond to a fcffward function block 840 arc removed. In a preferred 

direction 826o, a leftward direction 9Z6b, and a rightwaid embodiment, the nonconformal direction changes may be 

direction 826c, respectfully. Directions 826a-826c arc ^^^^ ^ returning to function block 816 and 

detod relauve to a preod^g ahgnment directioij 828 ^ nonconformed pixel 

between a current pixel 830 and an adjacent pixel 832 ,e\ ^ *^ . ... ^ . « . . . . . ^ 

representing the pre^g pixel in the chSn code. V'f'''^ ^ ^^^^^"^ ^ ^^^^ 

^ceding alignment dkection 828 may extend in any of ^ ^^^<>^^^ ,^^■1.^'^''''' embodunent such incurred 

the directionslw^own in HO. 24A, b^t is shown Jh a nonconformal dirc<^OD changes may be corrected m W 

specific orientation (i.e., right, horizontal) for purposes of ^ by checking for and co^g any incited noncon- 

iUustration. Direction 826fl is ddincd, therefore, as the same , direction chwigcs each tunc a nonconformal dircc- 

as direction 828. Directions 826i» and 826c differ from change is modified. 

direction 828 by leftward and rightward displacements of Function block 864 indicates that a Huffinan code is 

one pixel. generated from the resulting simplified chain code. Witfi 

B has been determined eiqwrimentally that slightly more chain codes 824a-824c corresponding to directions 

than 50% of chain codes 824 correspond to forward direc- 826A-826C that occur for about 50%, 25% and 25% of 

tion 826a, and slightly less than 25% of chain codes 824 ^ pixels in a contourt respective Huffinan codes of 0, 11, and 

correspond to each of directions 826^ and 826c. 10 are assigned. Such first order Huffinan codes allow c^ain 

Function block 836 represents an inquiry as to whether process 810 to represent contours at a bit rate of less than 1.5 

the next adjacent pixel in the contour conforms to one of bits per pixel in the contour. Such a hitrate represents 

directions 826. Whenever the next adjacent pixel in the approximately a 50% concession ratio improvement over 

contour conforms to one of directions 826, function block ^ conventional chain code processes. 

836 proceeds to function block 838, and otherwise proceeds It will be appreciated that higher order HufSnan coding 

to function block 840. could provide higher compression ratios. Higher order Huf- 

Function block 838 indicates that the next adjacent pixel finan coding includes, for exan^lc, assigning predetermined 

is assigned a chain code 824 conresponding to its direction ^ values to preselected sequences of first order Huffman 

826 relative to the direction 828 along whidi the adjacent codes, 
preceding pair of pixels are aligned. 

Function block 840 indicates that a pixel sequence con- Generation 

forming to one of directions 826 is substituted for the actual In some object based video coding methods, sprites are 

nonconformal pixel sequence. Based upon experimentation, generated for use in cormection with encoding determinate 

it has been determined that such substitutions typically will motion video (movie). Bitmaps arc accreted into bitmap 

arise in fewer than 0.2% of pixel sequences in a contour and series that comprise a plurality of sequential bitmaps of 

may be accommodated by one of six special-case modifi- sequential images from an image source. Accretion is used 

cations. to overcome the problem occluded pixels where objects 

FIG. 25C is a diagrammatic Tq>resentation of the six ^ or figures move relative to one another or where one figure 

special-case modifications 842 for converting non- occludes another similar to the way a foreground figure 

conformal pixel sequences to pixel sequences that conform occludes the background. For exano^le, when a foreground 

to directions 826. Within eadi modification 842, a pixel figure moves and reveals some new background, ^ere is no 

sequence 844 is converted to a pixel sequence 846. In each way to build that new background fi'om a previous bitmap 

of pixel sequences 844 of adjacent respective pixels X^, 45 unless the previous bitm^ was first enhanced by including 

A, B, the direction between pixels A and B does not conform in it the pixels that were going to be uncovered in the 

to one of directions 826 due to the alignment of pixel A subsequent bitmap. Hiis method cakes an incomplete image 

relative to the alignment of pixels and X^. of a figure and looks forward in time to find any pixels that 

In pixel sequence 844a, initial pixel alignments 850a and belong to the image but are not to be immediately visible. 

852a represent a nonconformal right-angle direction change. 50 Those pixels are used to create a conqx)site bitm^ for the 

Accordingly, in pixel sequence 846a, pixel A of pixel figure. With the composite bitmap, any future view of fee 

sequence 844a is omitted, resulting in a pixel direction 854fl figure can be created by distorting the composite bitmap, 

that conforms to pixel direction 826a. Pixel sequence modi- The encoding process begins by an operator identifying 

fications 8426-842/^ similarly convert nonconformal pixel the figures and the parts of the figures of a current bitmap 

sequences 844^-84^ to conformal sequences 8466^-846/ 55 from a current bitra^ series. 

respectively. Feature or distortion points are selected by the operator on 

Pixel sequence modifications 842 omit pixels that cause the features of the parts about which the parts of the figures 

pixel direction alignments that change by 90*^ or more move. A current grid of triangles is superinq^osed onto the 

relative to the alignments of adjacent preceding pixels XI parts of the current bitmap. The triangles that constitute the 

and X2. One effect is to increase the minimum radius of 60 current grid of triangles are formed by connecting adjacent 

curvature of a contour representing a right angle to over distortion points. 

three pixels. Pixel iXKMlifications 842 cause, therefore, a The distortion points are the vertices of the triangles. The 

minor loss of extremely fine contour detail. However, the cunent location of each triangle on the current bitmap is 

loss of such details is accq»table under most viewing con- determined and stored to the storage device. A portion of 

ditions. 6S data of the current bitmap that defines the first image within 

Fiinction block 860 represents an inquiry as to whether the current location of each triangle is retained for furtha 

there is another pixel in the contour to be assigned a ctain use. 
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A succeeding bitmap that defiiies a second inuge of the 
cunent bitmap secies is received from the image source, and 
the figures and the parts of the figure are identified by the 
operator. Next the cuircQt grid of triangles fi-om the cunent 
bitmap is supeiinq>osed onto the succeeding bitmap. The 
distortioo points of cuircnt grid of triangles arc realigned to 
coincide with the features of the coiiesponding figures on 
the succeeding bitmap. The realigned distortioii points form 
a succeeding grid of triangles on the succeeding bitmap of 
the second image. The succeeding location of each triangle 
on the succeeding bitmap is detcmiined and stored to the 
storage device. A portion of data of the succeeding bitmap 
that defines the second image within the succeeding location 
of each triangle is retained for further use, 

The process of determining and staring the current and 
succeeding locations of eadi triangle Is rqieated for the 
plurality of sequential bitmaps of the cuuent bitm^ series. 
When lhat process is completed, as average image of each 
triangle in the current bitmap series is detcmiined from the 
separately retained data. The average image of each triangle 
is stored to the storage device. 

During playback, the average image of each triangle of 
the current bitmap series and the current location of each 
triangle of the current bitmap arc retrieved from the storage 
device. A predicted bitmap is generated by calculating a 
transformation solution for transforming the average image 
of each triangle in the current bitmap series to die cuneot 
location of each triangle of the current bitmap and applying 
the transfonnation solution to the average image of each 
triangle. The predicted bitmap is passed to die monitor for 
display. 

In connection with a playback determinate motion video 
(video game) in which the images are determined by a 
controlling program at playback, a spnxc bitmap is stored in 
its entirety on a storage device. The sprite bitmap comprises 
a plurality of data bits that define a sprite image. The sprite 
bitmap is displayed on a monitor, and the parts of the sprite 
are identified by an operator and distortion points are 
selected for the sprite's parts. 

A grid of triangles is superimposed onto the parts of the 
sprite bitmap. The trian^es &at constitute the grid of 
triangles arc formed by connecting adjacent distortion 
points. The distortion points are the vertices of the triangles. 
The location of each triangle of the sprite bitmap is deter- 
mined and stored to the storage device. 

During playback, a succeeding location of each triangle is 
received from a controlling program The sprite bitmap and 
the succeeding location of each triangle on the sprite bitmap 
are recalled from the storage device and passed to the 
display processor. The succeeding location of eadi triangle 
is also passed to the display processor. 

A transformation solution is calculated for each triangle 
on the sprite bitmap. A succeeding bitmap is then generated 
in the display processor by flying the transformation 
solution of each triangle derived from the sprite bitmap the 
defines the sprite image within die location of each triangle. 
The display processor passes the succeeding sprite bitmap to 
a monitor for display. This process is repeated for each 
succeeding location of each triangle requested by the con- 
trolling program. 

As shown in FIG. 26, an encoding procedure for a movie 
motion video begins at step 900 by the CPU 22 receiving 
from an image source a current bidn^ series. The current 
bitmap series comprises a plurality of sequential bitmaps of 
sequential images. The current bitm^ series has a current 
bitmap that comprises a plurality of data bits which define a 
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first iniage from the image source. The first image comprises 
at least one figure having at least one pait 

Proceeding to step 902, the first image is displayed to the 
operator on the monitor 28. From the monitor 28, the figures 

^ of the first image on the current bitmap are identified by the 
operator. The parts of the figure on the current bitmap are 
then identified by the c^jcrator at step 904. 

Next, at step 906, the operator selects feature or distortion 
points on the cuircnt bitmap. The distortion points arc 
selected so that the distortion points coincide with features 
on the bitmap whae relative movement of a part is likely to 
occur. It will be understood by those skilled in the art that the 
figures, the parts of the figures and the distention points on 
a bitmap may be identified by the computer system 20 or by 
assistance from it It is prefeired, however, that the operator 
identify the figures, the parts of the figures and the distortion 
points on a bitmap. 

Proceeding to step 908, a current grid of triangles is 
superimposed onto the parts of the current bitmap by the 
computer system 20. With reference to FIG. 27 A, the cuirent 
grid comprises triangles formed by connecting adjacent 
distortion points. The distortion points form the vertices of 
the triangles. More specifically, the first image of the current 

^ bit map comprises a figure, which is a person 970. The 
person 970 has six parts corresponding to a head 972, a torso 
974, a right arm 976, a left arm 978, right leg 980, and a left 
leg 982. Distortion points are selected on each part of die 
person 970 so that the distortion points coincide with 

^ features where relative movement of a part is likely to occur. 
A current grid is superiiiqx>$ed over each part with the 
triangles of each current grid formed by connecting adjacent 
distortion points. Thus, the distortion points form the verti- 
ces of the triangles. 

35 At step 910, the computer system 20 determines a current 
location of each triangle on the current bitmap. The cuirent 
location of each triangle on the cuircnt bitmap is defined by 
the location of the distortion points that form the vertices of 
the triangle. At step 912, the current location of each triangle 

4Q is stored to the storage device. Aportion of data derived from 
the current bitmap that defines the first image within the 
current location of each triangle is retained at step 914. 

Next, at step 916, a succeeding bitmap of the current 
bitmap series is received by the CPU 22. The succeeding 

45 bitmap comprises a plurality of data bits which define a 
second image of the current bitmap series. The second image 
may or may not include figures that correspond to the figures 
in ^e first image. For the following steps, the second image 
is assumed to have figures that corresponds to the figures in 

50 the first image. At step 918, the cuirent grid of triangles is 
superin^osed onto the succeeding bitmap. The second 
image with the superin^osed triangular grid Is displayed to 
the operator on the monitor 28. 
At step 920, the distortion points are realigned to coincide 

S5 with corresponding features on the succeeding bitmap by the 
operator with assistance from the con^uter system 20. The 
computer system 20 realigns the distoition using block 
matching. Any mistakes arc corrected by the operator. Witti 
reference to FIG. 27B, the realigned distortion points form 

60 a succeeding grid of triangles. The realigned distortion 
points are the vertices of the triangles. More spedficaUy, the 
second image of the succeeding bitmap of person 200 
includes head 972, torso 974, right aim 976, left arm 978, 
right leg 980, and left leg 982. In the second image, however, 

65 the right arm 980 is raised. The current grids of the first 
image have been superin^>osed over each part and their 
distortion points realigned to coincide with corresponding 
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features oa the second image. The realigiicd distortion points bitmap that comprises a phiralit/ of data bits which define a 

define succeeding grids of triangles. The succeeding grids first image from the image source. The first image conquises 

comprise triangles formed by connecting the realigned dis- at least one figure having at least one part, 

tortion points. Thus, the reaUgned distortion point form the At step 1002, the average image of each triangle of the 
vertices of the triangles of the succeeding grids. 3 current bitmap series is retrieved firom the storage device. 

Proceeding to step 922« a succeeding location of each The average image of each Oiangle is then passed to a 

triangle of the succeeding bitnup Is determined by the display processor (not shown) at step 704. It will be &ppi^ 
computa system 20. At step 924, the succeeding location of dated that computer system 20 (FIG. 1) can optionally 

each triangle on the succeeding bitmap is stored the storage include a display processor or other dedicated components, 

device. A portion of data derived firom the succeeding Proceeding to step 1006, the current location of each triangle 

bitmap that defines the second image within the succeeding on the current bitmap is retrieved firom the storage device, 

location of each triangle is retained at step 926. Step 926 The current location of each triangle is passed to the display 

leads to decisional st^ 928 where it is determined if a next processor at step 1008. 

succeeding bitmap exists. Next, an aflfine transfcraation solution for transfOTming 

If a next succeeding bitmap exists, the YES branch of the average image of each triangle to the current location of 

decisional step 928 leads to step 930 where the succeeding each triangle on the cuirent bitmap is calculated by the 

bitmap becomes the cuirent bitmap. Step 930 returns to step display processor at step 1010. Proceeding to step 1012, a 

916 where a succeeding bitmap of the current bitsiap series predicted bitmap is generated by the display processor by 

is received by the CPU 22. If a next succeeding bitmap does applying the transformation solution for transforming the 
not exist, the NO branch of decisional step 928 leads to step ^ average image of each triangle to the current location of each 

932 where an average image for each trian^c of the cuacnt triangle on the current bitmap. 

bitmap series is determined. The average image is the At step 1014, a coiTcction bitmap for the current bitmap 

median value of the pixels of a triangle. Use of the average is retrieved from the storage device. The correction bitmap 

image makes the process less susceptible to degeneration- is passed to the display processor at step 716. A display 
Proceeding to step 934, the average image of each triangle " bitmap is then generated in the display processor by over- 

of the current bitmap series is stored to the storage device. laying the predicted bitmap with the correction bitmap. The 

Next, at step 936, the current location of each triangle on display processor retains a copy of the average image of 

the current bitmap is retrieved from the storage device. An cadi triangle and passes the display bitmap to the frame 

affine transformation solution for transforming the average buffer for display on the monitor. 

image of each triangle to the current location of the triangle Next, at decisional step 1020, it is determined if a sue- 
on me current bitmap is then calculated by the con^juter ceeding 30 bitmap of the current bitmap scries exists. If a 
system 20 at step 938. At step 940, a predicted bitmap is succeeding bitmap of the current bitmap scries exists, the 
gentrated by applying the transformation solution of the YES branch of decisional step 1020 leads to step 1022. At 
average inaage of each triangle to Ac current location of each step 1022, the succeeding bitmap becomes the current 
triangle on the current biunap. The predicted bitmap is bitmap. Step 1022 returns to step 1006 where the location of 
compared with the current bitm^ at step 942. eadi trian^e on the current bitmap is retrieved from the 

At step 944 a correction bitmap is generated. The cor- storage device, 

rected bitm^ comprises the data bits of the current bitmap Returning to decisional step 1020, if a succeeding bitmap 
that were not accurately predicted by the predicted bitmap. ^ of the current bitmap scries does not exist, the NO branch of 

The corrected bitmap is stored to the storage device at step decisional step 1020 leads to decisional step 1024. At 

948. Step 948 leads to decisional step 950 where it is decisional step 1024. it is determined if a succeeding bitmap 

dctcnnined if a succeeding bitmap exists, series exists. If a succeeding bitmap scries does not exist, 

If a succeeding bitmap exists, the YES branch of deci- then the process is finished and the NO branch of decisional 
sional step 950 leads to st^ 952 where the succeeding 45 step 1024 leads to step 1026. If a succeeding bitmap series 
bitmap becomes the current bitmap. Step 952 renirns to step exists, the YES branch of decisional step 1024 leads to step 
936 where the current location of each triangle on the cuncnt 1028. At step 1028, the succeeding bitmap series becomes 
bitmap is retrieved from the stOTage device. If a next the current bitmap series. Step 1028 returns to step 1000. 
succeeding bitmap does not exist, the NO l^anch of deci- 
sional step 950 leads to decisional step 954 where it is Representation and Encoding of General Arbitraty 
detenmined if a succeeding bitmap scries exists. If a sue- Shapes 

ceeding bitmap scries docs not exist, encoding is finished HG. 29 is a diagrammatic representation of a soHd binary 

and the NO branch of decisional step 954 leads to step 956. arbitrary feature or shape UOO representing a binary mask of 

If a succeeding bitmap scries exists, the YES branch of an arbitrary object included in a frame of a video image 
decisional step 954 leads to step 958 where the CPU 22 55 sequence. As described above, each frame of a video image 

receives the succeeding bitmap sexies as the current bitmap sequence typically includes multiple objects corresponding 

series. Step 956 returns to step 5W2 where tfie figures of the to multiple image features such as characters, props, and 

first image of the current bitmap scries is Identified by the background. The configuration of solid shape 1100 is arbi- 

op®™^^* trary to rq)reseiit any such object having a solid cr continu- 

The process of FIG. 26 describes generation of a sprite or 60 ous interior, 

master object 90 for use by encoder process 64 of FIG. 3. As a binary representation relative to its background 

The process of utilizing master object 90 to form predicted 1102, solid shape 1100 cotrcsponds to a mask, such as those 

objects 102 is described with reference to FIG. 28. described hereinabove with respect to FIGS. 2A, 2B, and 

As shown in FIG. 28, the procedure begins at step lOCK^ 3A, by which objects are identified and encoded. Solid shape 
witti a current bitmap series being retrieved. The current 65 UOO is characterized by a continuous outer contour or 

bitmap series con^sises a plurality of sequential bitm^s of boundary 1104 and a uniform or single-state interior 1106 

sequential images. The current tntmap series has a current within boundary 1104. Solid sh^ UOO includes no dis- 
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connected or embedded poitioas of different binary states* 
Solid shape 1100 is capable of being compressed or encoded 
accurately with respect to its boundary 1104 by conventional 
contour coding tediniques sudi as chain coding or polygo- 
nal contour approximation, or by the simplified chain encod- 
ing process described hereinabove with reference to FIGS. 
25A-25C. 

FIG. 30 is a diagrammatic rqrcsentation of a general 
binary arbitrary feature or shape 1110 r^resenting a binary 
mask of an arbitrary object included in the fraroe of a video 
image sequence. General shape 1110 preferably corresponds 
to a binary mask distinct from its background 1112 by which 
objects are identified and encoded. The configuration of 
genaal shape 1110 is general in that it represents generally 
any object, including objects having discontinuous or 
embedded regions or con^>onents within their interiors. 

In this regard, solid shape 1100 represents a simplified 
subset of general shape 1110. 

General shape lUO includes multiple continuous con- 
tours or boundaries 1114 that are disconnected or enclosed 
withio each other. FIG. 30A shows a first set of disconnected 
boundaries 1114a-1114c corresponding to a first hierarchi- 
cal level, a second set of disconnected boundaries 
1114^-1114^ couesponding to a second hierarchical level, 
and a third set of disconnected boundaries 1114g and 1114^ 
corresponding to a third hierarchical level. Boundaries 
1114a-1114h bound or encompass corresponding uniform or 
single-state oon^nents 1116a-1116A. Accordingly, general 
shdpc lUO includes disconnected components (e.g., 1114a 
and 1114c) and embedded components (e.g., 1114^ and 
1114^) within host components (e.g., 1114a and lll4e); the 
embedded components being of different binary states than 
their host conqx)nents. Embedded components are analo- 
gous to holes or islands within host components correspond 
to islands or holes, respectively. 

Accurately recognizing and encoding the disconnected or 
embedded components of solid shape 1110 provide 
improved video compression l>ecause such general shapes 
CQirespond better to many objects commonly found in 
general video image sequences. The disconnected or embed- 
ded components of general sh^e 1110 cannot be repre- 
sented by some conventional shape encoding tediniques and 
are represented inefficiently by other techniques. As a 
consequence, such general shapes conventionally are sim- 
plified to ignore embedded components, which can intro- 
duce significant encoding errors during video compression. 

FIG. 31 is a functional block diagram of a hierarchical 
decomposition and encoding process 1130 capable of accu- 
rately representing general arbitrary shape 1110 with its 
disconnected and embedded components 1116a-1116^. 
Hierarchical process 1130 automatically decomposes gen- 
eral binary arbitrary shapes into distinct component masks 
that have continuous boundaries and no embedded compo- 
nents of contrasting binary states. Process 1130 is hiaar- 
chical in that embedded components are decomposed fi'om 
host components iteratively to form hierarchical levels of 
conq)onent masks. Each con^onent mask is capable of 
being compressed or encoded accurately with respect to its 
boundary by conventional contour coding techniques sudi 
as chain coding or polygonal contour approximation, or by 
the simplified chain coding process desoibed hereinabove 
with reference to FIGS. 25A-25C. 

Hierarchical process 1130 receives binary shape data 1132 
corresponding to general arbitrary shape 1110, which may 
include simple solid sh^>es and general shapes having 
disconnected ch* embedded coiiq)onents. For purposes of 
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explanation, hierarchical representation process 1130 is 
described with idaeacc to general arbitrary shape 1110 
with its embedded and disconnected components 
1116a-lll<i/(, but is similady applicable to single shape 
5 1100. 

Process block 1134 indicates that a bounding box 1136 of 
pixels (FIG. 30B) is defined about and encompasses com- 
ponents 1116o-1116/r of general arbitrary shape 1110. 
I^eferably, bounding box 1136 is a right regular array of 

Q pixels with dimensions sdected automatically or by a uscl 
It will be ^>preciated that bounding box 1136 as used with 
respect to process 1130 preferably is oversized relative to the 
con^nents lU6a-lU6/t of general arbitrary shape 1110. In 
some applications, a bounding box is deemed as being fitted 

^ closely to an enclosed feature. Bounding box 1136 is pref- 
erably oversized to assure that all components of a shape arc 
enclosed. 

Process block 1140 indicates that an initial pixel 1142 
within bounding box 1136 and corresponding to background 
^ 1112 is sought. In the preferred binary rq)resentatian, back- 
ground 1112 is of a known binary state. Initial pixel 1142 is 
sought initially at a selected comer of bounding box 1136 
(e.g., the upper left comer shown in FIG. 30B), If that 
location corresponds to general shape 1110 radier than 
^ background 1112, a search is commenced successively at the 
remaining corners and along the boundaries of bounding box 
1136 to identify an initial pixel 1142 corresponding to the 
background 1112. 

Decision block 1144 represents an inquiry whether an 
3Q initial pixel 1142 is identified along the boundary of bound- 
ing box 1136 and corresponding to background 1112. When- 
ever such an initial pixel 1142 is identified, decision block 
1144 proceeds to process block 1146. Whenever no pixel 
along the boundary of bounding box 1136 corresponds to 
33 background 1112, decision block 1144 proceeds to process 
block 1148. 

Process block 1146 indicates that all pixels of the binary 
state corresponding to background 1112 and connected 
together with initial pixel 1142 are assigned the opposite 

40 binary state. As a result, bounding box 1136 is **filled" 
around major objects 1116a-1116c and forms shapes 
con^jlcmentaiy to major objects 1116a-1116c. This filling 
of bounding box 1136 may be performed by any conven- 
tional filling technique such as region grow, whidi is 

43 explained in Computer Gmphics: Principles and Practice, 
2d cd., Foley et al., Addison-Wesley Publishig Co,, N.Y:, 
(1991). As shown in FIG. 30C, the filling of background 
1112 in bounding box 1136 leaves unfilled complementary 
connected components 1142a-1142c conesponding to 

50 respective major objects 1116a-1116c. Connected compo- 
nents 1142a-1142c encompass all objects embedded within 
majcff objects 1116a-1116c a provide a first hierarchical 
decomposition of general object 1110. 
Process block 1148 indicates that the pixels within bound- 

55 ing box 1136 of the same binary state as, and connected to, 
the pixels along the boundary of bounding box 1136 are 
assigned the opposite binary state. It will be appreciated that 
the pixels filled by this process block relate not to back- 
ground 1112, but rather an object or objects (not shown) that 

60 extend to the boundary of bounding box 1136. As a result, 
bounding box 1136 is **filled" around objects embedded 
within one or more host objects and forms shapes comple- 
mentary to the embedded Objects. This filHiig of bounding 
box 1136 may be performed by the same filling technique 

65 used in coimcction with process block 1146. 

Process block 1150 indicates that connected components 
fom^ by process blocks 1146 and 1148 are identified and 
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filled, ^th refereoce to the connected components fonned 
by process block 1146, for example, connected components 
1142a-1142c are identified by thcii contrasting binary state 
from that of the filled baclcground 1112 and preferably are 
filled to form solid masks conesponding to respective major 5 
objects 1116a-1116c. The solid masks concspooding to 
connected components 11420-1142c provide a basis for 
identifying and processing similarly objects embedded 
within major objects 1116a-lll6<:. 

Process block 1152 indicates that a boundary or contour 
of each of the connected cnnponents identified by process 
block 1150 is encoded or compressed by a contour coding 
technique such as conventional chain coding or conventional 
polygonal contour approximation, or preferably, by the 
simplified chain coding process described hereinabove with 
references to FIGS. 25A-25C. It will be appreciated that 
ead) of the connected coiiq>oneDts (e.g., connected compo- 
nents 11420-1142c) is effectively a sin:q>le binary object 
capable of being represented accurately by such contour 
encoding techniques. Complementary components 
1142a-1142c represent one level of general arbitrary object 20 
decomposition tiiat accurately represents objects at a com- 
mon level of hierarchical decon^x>sition. Subsequent itera- 
tions of process 1130 provides analogous representations of 
successively embedded objects. 

Difference block 1154 indicates that a logical difference is 2i 
taken between the complementary componeuts identified in 
accordance with process block 1150 and the coire^nding 
oi)jects in the (»iginal binary shape data 1132. The difference 
is determined on a pixel-by^ixel basis. F<x example, the 
difference between majcs* objects 1116a-1116c and the solid 30 
masks formed from respective complementary components 
1142ci-1142c identifies any discontinuous objects embedded 
within objects 1116a-1116c. FIG. 30D is a diagrammatic 
representation of the resulting difference showing that 
within major object 1116a (shown in outline for reference 35 
purposes) embedded objects 1116^ and 1116« are identified 
and that within major object 11166 (shown in outline for 
reference purposes) embedded object 1116/ is identified. 
FIG. 30D also demonstrates that the absence of a difference 
between major object 1114c and complementary component 40 
1142c indicates that no objects are embedded therein. As a 
result, encoding boundary 1116c of object 1114c completely 
describes and represents in a compressed formal object 
U14c. 

Difference block 1154 identifies discontinuous embedded 45 
objects (e.g. 1116^, 1116?, and 1116/), which arc delivered 
to process block 1132 for processing in the same maimer as 
were major objects 1116a-1116c. Moreover, each succes- 
sively embedded Layer cf objects, such as objects 1116^ and 
1116A within object 1116e, also is processed successively in 50 
this manner. Ihus, successively embedded objects or layers 
are processed hierarchically by this method to encode accu- 
rately general arbitrary binary shapes. The difference opera- 
tion of difference block 154 functions to identify discon- 
tinuous embedded objects. This function could be achieved 55 
alternatively by assigning the other binary state to the 
complementary oon^nents and summing them with the 
original binary shape data. 

It will be appreciated that as binary objects, successively 
embedded discontinuous components alternate between first 60 
and second binary states. For reference purposes, objects 
identified by even-numbered operations of difference block 
1154 (e.g. 0, 2 . . . ) arc referred to as ''islands" and include 
in FIG. 30A objects 1116a-1116c, 1116^, and 1U6A, Objects 
identified by odd-numbered operations of difference block 65 
1154 (e.g. 1, 3 ... ) are referred to as "holes" and include in 
HG. 30A objects 1116^, lU6e, and 1116/ 
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Reconstruction or reccn:q)osition of general binary aiti- 
trary shape 1119 &om the contour encoded objects identified 
hierarchically by encoding process 1130 may be pcxfoimed 
hierarchically according to the sequence in which successive 
islands and holes are identified. Each successive hierarchical 
level is overlaid on a previous, hierarchically higher level 
For example, complementary conqjonents 1142a-1142c 
would initially be decoded or decompressed from their 
contour encoded farmats, as is known in the art or described 
above. Subsequently, complementary components 
1142J-114^ corresponding to holes 1116^^1116/ would be 
decoded or decompressed and overlaid on complementary 
components 1142a and 1142^. Finally, complementary com- 
ponents 1142g and lX42h corresponding to embedded 
islands 1116^ and 1116A would be decompressed and over- 
laid onto the reconstructed shape. As a result, general 
arbitrary shape 1110 corresponding to a binary mask may be 
accurately encoded and decoded for coix^ressed storage or 
transmission. 

FIG. 32 is a functional block diagram of an encoding 
process for representing non-binary object information such 
as object transparency data, which is sometimes referred to 
as an alpha channel As is known in the art, each pixel of a 
video image has a pixel value corresponding to predefined 
image characteristics. Frequently, pixels are assigned color 
component values corresponding to red, green, and blue- 
color components that together provide a substantially full 
color range. Each color component could be represented, for 
example, by an 8-bit digital value. Alternatively, pixel values 
can be represented by a YUV uniform color space in which 
Y represents luminance and U and V represent cfaromatidty, 
as is known in the art Each of such Y, U, and V color space 
components also could be represented by 8-bit digital val- 
ues. 

Id addition to such color space representations for the 
pixels of an image, some object based video image rqpre- 
sentations include a transparency or ^'alpha'* channel that 
represents the relative transparency of the pixels correspond- 
ing to a selected object Alpha channels conunonly are used 
in video coding or compression, as well as coniputer 
graphics, image composition, eta On a normalized scale, for 
example, an alpha or transparency value of 0 could represent 
con^plete transparency and correspond to an object (e.g. 
bad^ground) over which any other object with a non-zero 
transparency value would be rendered. In contrast, a nor- 
malized transparency value of 1 could represent complete 
opacity such that a corresponding object would be rendered 
ova any other object in an image. It will be appreciated that 
such transparency values can be represented by at least 8-bit 
and frequently 12- or 16-bit, digital values and that the 
relative transparency values of overlapping objects is used to 
rqircscnt and render overlapping objects. 

Encoding or compressing video data that includes a 
transparency channel requires that the transparency channel 
also be encoded or compressed. 

However, acceptable encoding cf a transparency channel 
requires that the boundaries of transparency representation 
be accurately encoded and decoded. Erroneous representa- 
tions of the transparency channel ttoundaries of an object or 
objects creates discernible and undesirable discontinuities in 
a decompressed or regenerated image. 

FIG. 32 is a functional block diagram of an encoding 
process 1160 for representing non-binary object 
information, such as object transparency data, so as to 
maintain accurate representations of object boundaries. 
Encoder process 1160 provides accurate transparency data 
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boundary identification and encoding by hierardiical encod- 
ing process 1130 (HG. 31). In addition^ encoding process 
1160 utilizes precompression extrapolation method 400 
(FIGS. 17A and 17B) for extrapolating transparency values 
for objects of arbitrary configuration to a predefined con- 
figuration to facilitate compression cr encoding in a con- 
ventional manner, sudi as by discrete cosine transform 
(DCT) or lattice wavelet compression, as described above. 

This combination of hierarchical encoding process 1130 
and precompression extrapolation method 400 allows trans- 
parency data to be encoded efficiently while maintaining 
highly accurate representations of transparency data bound- 
aries. Moreover, it will be appreciated that encoding process 
1160 would be similarly ^plicabie to other multi-value 
object data types for which accurate boundary representa- 
tions and compression efficiency are necessary or desirable. 

Encoding process 1160 receives multi-value transparency 
data 1162 corresponding to a region of a video image frame. 
Typically, transparency data 1162 would correspond to one 
or more objects, some of which may be partly or co^^)letcly 
overlapping others. Different transparency values typically 
would be associated witfi different ones of the objects 
according to the relative transparency or opacity of the 
objects. 

lYocess block 1164 indicates that a threshold filter is 
applied to the transparency data. The threshold filter typi- 
cally would have a relatively low, sometimes zero, threshold 
value to distinguish highly or completely transparent objects 
(e.g., backgroimd) from other objects. The threshold filter of 
process block 1164 provides a binary image representation 
that can include general arbitrary shapes of the type 
described hereinatx>ve. 

Process block 1168 indicates that the binaiy transparency 
data are applied to hierarchical encoding process 1130. 
Encoder process 1130 hierarchically decon^oses and 
encodes the binary transparency data to provide precise 
encoded representations of the corresponding boundaries of 
the transparency data, as described above with reference to 
FIG. 31. 

Process block 1170 indicates that the transparency data 
1162 received by encoding process 1160 are extrapolated to 
a predefined configuration to facilitate compression, 
ftefcrably, the transparency data are extr^lated by pre- 
compression extr^lation method 400, described herein- 
above with reference to FIGS. 17A and 17B, and the 
predefined configuration of extrapolation block boundary 
406 (FIGS. 18) corresponds to bounding box 1136 of 
encoding process 1130. 

Process block 1172 indicates that the cxtrs^lated trans- 
parency data are encoded by an intraframe encoding process 
such as DCT or lattice wavelet encoding. It will be 
^predated, however, that interframe encoding as described 
above with reference to process 64 can also be applied to the 
transparency data, resulting in a residual signal that prefer- 
ably would be encoded by DCT or lattice wavelet encoding. 

Encoding process 1160 provides as compressed or 
encoded data for storage or transmission an encoded bound- 
ary representation at process block 1168 and an intra-frame 
encoded representation of the transparency data at process 
block 1172. Decoding of this information includes conven- 
tional intra-frame decoding of the transparency value data 
(e.g. DCT or wavelet), decoding the boundary infcrroation 
corresponding to the binaiy transparency objects identified 
by the threshold filter of process block 1164, and ^plying 
the decoded boundary information as a mask to the decoded 
transparency value information to r^esent reconstituted or 
decompressed transparency data. 
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Skipping of Transparent Transformation and Sub- 
l^sfonnatioa Blocks 

Hie object-based video coding methods described above 
code shape and texture independently. The shape informa- 
tion can be used to increase coding efficiency because it 
enables coders/decoders to determine to skip coding or 
decoding of transparent transformation blocks. Video coding 
methods sometimes encode transfocxnation blocks in smaller 
blocks, such as 8x8 pixel blocks. The shape information can 
' also be used to determine when to skip coding or decoding 
these smaller blocks. In the description to follow, the trans- 
formation blocks arc refeired to as **maCToblocks" while the 
smaller blocks within each macroblock are called *'blocks.** 
To avoid confusion between transformation blocks and the 
' smaller blocks that they are comprised of, we sometimes 
refer to the smaller blocks as sub-transformation blocks. 

In an object-based coding method, the video objects in a 
sequence of video frames are coded separately and the 
^ resulting compressed video data is combined into a bit- 
stream. To deconqiress this bitstream, a decompressor 
(cither hardware or software) separates the bitstrcam into 
separate objects, decodes object-based data to reconstruct 
the objects in the frames, and composites the objects to form 
the original sequence of video frames. FIGS. 33 and 34 are 
general block diagrams illustrating the structure of an 
object-based video encoder and decoder, respectively. 

FIG. 33 illustrates an example of the structure of an 
object-based video encoder. The input to the encoder 1500 
typically includes a sequence of video frames comprised of 
natural images, synthetic images (e.g., the ou^ut of a 3D 
rendering system or computer generated graphics), or a 
combination of both. 
The object definition block 1502 of the encoder deter- 

3 j mines how to separate this input video sequence into objects. 
The object definition process generally includes identifying 
separate objects in the input video sequence and defining the 
shape of these objects. At the end of this process, an object 
has shape information and is associated with a bounding 

4Q rectangle that encloses the object. Each of the objects 
represents parts of the image frames in the video sequence, 
and these parts arc represented by image data such as an 
array of pixel values, where each pixel value has color 
components (YUV or RGB, for cxanQ)le). The shape infor- 

43 mation for an object describes the boundary or ^^contour'* of 
the object within its bounding rectangle. 

The shape information is either generated by segmenta- 
tion or is predefined, as in the case of synthetic objects that 
already have an alpha {Hane. The shape information is 

50 typically represented by a mask such as an array of alpha 
values (e.g., 8 bit grey scale alpha) associated with a 
synthetic object or a binary mask generated during the 
segmentation process. Each object can have an arbitrary 
sh^e. One way to generate shape inf^mation for natural 

55 image video is to use die well-known "bhie screen" tech- 
nique. In this approach, an object or objects are filmed in 
front of blue screen. The blue background in each frame can 
then be used to generate the shape information of the object 
for each frame: the blue region in each frame rq>resents the 

50 area outside an object, while the non-blue area represents the 
object . 

After the object definition phase, the encoder separately 
codes objects as illustrated in the coding units 1504-1508 
shown in HG. 33. These coding units 1504-1508 encode the 
65 shape, motion and texture for each object 

The texture data of an object represents either: 1) an array 
of color intensity values such as YUV cb- RGB for intra- 
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frame coding; or 2) an ezror signal representing the differ- 
ence between color values in a ptredicted object and the 
actual object for intcr-framc coding. The coding units 
15O4-150S use a coding metiiod sudi as wavelet or DCt 
coding to code inter and intraframe texture data. 

While we provide specific exanoples of shape, motion, and 
texture coding, the specific coding methods are not critical 
to the invention and conventional shape, motion and texture 
coding methods can be used. 

The ou^t of the coding units 1504-1508 is combined to 
form a bitstream of compressed video data. In FIG. 33, the 
process of combining coded object data is represented by the 
tmiltiplexer 1510, which receives the ou^ut from the coding 
units and combines the coded objects into the bitstream 
1511 

FIG. 34 is a block diagram illustrating an object-t>ased 
video decoder. The decoder receives a bitstream 1520 of 
encoded video data and separates this bitstream into sepa- 
rately encoded objects as illustrated by the demultiplexer 
1522. 

The decoder separately decodes the objects In decoding 
blocks 1524-1528. For intra-frame encoded objects, the 
decoding blocks 1524-1528 decode the objects shape and 
texture. For inter-frame objects, the decoding blocks decode 
sh^ texture and motion for each object. The encoder then 
composites reconstructed objects in the con4x>sitor 1530 to 
produce reconstructed frames in a video sequence 1532. 

As an example of object-based coding, FIG. 35 illustrates 
a frame of video in terms of the objects 1540-1544 that 
make up the frame. In this cxan^lc, the fr^e 1538 is 
segmented into 3 objects: a person 1540, a spaceship 1542, 
and the background with landscape 1544a, tree 15446, and 
sky 1544c. For simplicity we refer to the background 
genarally using reference numeral 1544. 

FIG. 35 shows the object representing a person 1540 in 
expanded form to show how this portion of the image is 
divided iato transfocmatioD blocks. As part of die object 
definition process, the encoder computes a bounding rect- 
angle 1546 for the object 1540. To code the object using 
transformation blocks, the encoder expands the bounding 
rectangle such that the rectangle is an integer multiple of 
transformation blocks (1548-1552) in both the vertical and 
horizontal direction. The transformation blocks 1548-1552 
in this example are sometimes referred to as macroblocks. 
Each macroblock is further divided into sub-transformation 
blocks, referred to as '•blocks." 

During the coding process, the encoder codes the shape of 
the objects (e.g., 1540, 1542) separately from the objects* 
texture and motion data. 

In FIG. 35 for example, the shape of the object repre- 
senting the person 1540 is represented by a mask, and this 
mask is coded and decoded squaratdy from the object's 
texture or motion data. 

Since the object 1540 representing the psson in video 
frame 1538 is separated from the other objects in the frame, 
die region inside the bounding box 1546 is likely to have 
some transparent macroblocks and blocks. The overhead 
and number of bits needed to encode the object's texture and 
motion data can be reduced by using shape to determine 
which transformation blocks (e.g., macroblocks) and sub- 
transfonnation blocks (e.g. blocks) are transparent (i.e. not 
covered by the object 1540). Once these transparent mac- 
roblocks and blocks are identified, the coder and decoder can 
skip coding for these macroblocks or blocks. Skipping of 
transparent transformation blocks applies when the entire 
transformation block is transparent Skipping of transparent 
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sub-transformation blocks applies to transformation blocks 
partially covered by an object. A '"partially covered" mac- 
roblock may include one or more transparent blocks and one 
or more blocks covered by a portion of an object. 

5 An exanq)le of a transparent macroblod^ is maaoblock 
1548, which lies entirely outside object 1540. An example of 
a partially covered macroblock is macroblock 1550, which 
includes transparent blocks 1554-1558 and partially cov- 
ered block 1560 covered by a portion of the object 1562. 

JO Before describing skipping of transparent transformation 
blocks, we describe object-based coding in more detail This 
will provide a context for transparent block skipping, which 
is described in more detail bdow. 

FIGS. 36 and 37 are block diagrams illustrating an 
object-based 25 encoder and decoder in mcsc detail The 
object coder shown in FIG. 36 includes three basic parts: a 
shape coder 1580, motion coder 1582 and texture coder 
1584. 

The shape coder 1580 reads an objects shape information 
such as an object mask and encodes it The shape coder can 

^ use a variety of shape coding methods including the arbi- 
trary shj^ coding methods described above. 

The motion coder 1582 performs motion estimation and 
motion compensation on an object It analyzes an object in 
position in a current frame and one or mart other frames 

^ (previous or subsequent frames), and computes motion data 
defining how this object moves from frame to frame. Motion 
data can include a series of motion vectors and/or transform 
coefEdents as described in detail above. The motion esti- 
mation data generated by the motion coder 1582 forms part 

30 of the bitstream representing the coinpressed video sequence 
and is used to predict the object's motion. The motion coder 
also computes error signals for inter-frame coded object 
data. ThtsG error signals represent the difference between a 
predicted object, transformed using the motion data, and the 

35 actual object for the current frame. The error signals are fed 
into the texture coder. 

The texture coder 1584 codes tiie objects texture for both 
intra and inter-framed object data. In the intra-frame case, 
texture data includes an array of color values, whereas in the 

4Q inter-frame case texture date comprises the error signals 
produced from motion con^nsation. A variety of still 
image compression techniques can be used to compress inter 
or intra-frame texture data. Id the example illustrated in FIG. 
35, each macroblock is divided into 8x8 pixel blocks, which 

4j can be compressed using conventional DCT or wavelet 
coding methods. 

The multiplier 1586 shown in FIG. 36 combines the 
coded shape, motion and texture data for an object into a 
bitstream of conq}res$ed video data. 

so Coded shape, motion, and texture data for each of the 
objects in the video sequence are combined to form the 
bitstream r^resenting the video sequence. FIG. 37 illus- 
trates a general block diagram of a decoder in an object- 
based video coding system. The demultiplexer 1610 reads 

$5 the bitstream 1612 of coded video data and separates it into 
encoded shape data, motion data and texture data for the 
objects in the video sequence. The shape decoder 1614 
decodes the shape information for objects. The sh^ infor- 
mation is decoded first so that the decoder can use it to 

60 identify transparent macroblocks and blocks associated with 
eadi object As part of the process of reconstructing an 
object for a frame, the decoder applies the shape inf onuation 
to the object's texture data to remove decoded texture data 
falling outside the object's boundary. 

65 The texture decoder 1616 decodes both intra-frame and 

. inter-frame coded texture data f<r objects in the video 
sequence. 
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The motion decoder 1618 decodes the motion data and FIG, 39 is a block diagram illustrating a dccoda for an 

pcifonns motion conqjcnsation to reconstruct an object The object-based video coding method. A demultiplexer 1660 

motion decoder traasforms a previously reconstructed object receives a bitstrcam representing a compressed video 

using decoded estimation data to compute a p-cdicted sequence and sq}arates sh^es, motion aad texture encoded 
object It dien combines the decoded error values from the 5 data on an object by object basis. Shape decoding block 

texture decoder with the predicted object to compute a 1664 decodes the shape or contour for the current object. To 

reconstructed object for the cunent frame. The output of the accomplish this, it employs a shape decoder that implements 

decoder in this reconstructed object 1620. the inverse of the sh^c encoding method used in the 

FIG. 38 is a block diagram illustrating a more spediic encoder of FIG. 38. The resulting shape data is a mask, such 

implementation of an object-based video encoder. The input as a binary alpha plane or gray scale aJ^ha plane rc|>rcsent- 

1630 to the encoder includes a series of objects, their shape ing the shape of the object. 

information and bounding rectangles. The shape The motion decoding blodc 1666 decodes the motion 
infonnation, therefore, is available before the encoder codes information in the bitstream. The decoded motion informa- 
texture or motion data. This enables the encoder to deter- tion includes motion data such as motion vectors for mac- 
mine which maaoblock and blocks wiAin the object's roblocks blocks or transform coefficients, depending on the 
bounding rectangle are transparent and can be skipped. type of estimation method used in the encoder. The motion 
The shape coding block 1632 receives the definition of an decoding block 1666 provides this motion information to the 
object induding its bounding rectangle and extends the motion con^)ensation block 1668, and the motion compen- 
bounding rectangle to integer multiples of macroblocks. The sation block 1668 applies the motion data to previously 
shape information for an object in this irnplementation reconstructed object data 1670. 

comprises a mask or "alpha plane." The shape coding block The texture decoding block 1674 decodes error signals for 

1632 reads this mask and compresses it interframc coded texture data and an airay of color values 

Motion estimation block 1634 reads an object including for intra-frame texture data and passes this infonnation to 

its bounding rectangle and a previously reconstructed object the block labeled reconstnicted object For inter-frame 
1636 and computes motion estimation data used to predict ^ coded objects, the reconstructed object block 1672 applies 

the motion of an object from one frame to the next. The the error signal data to the predicted object output from the 

motion estimation block 1634 applies the motion estimation motion compensation block to compute the reconstructed 

method described above or a conventional motion estima- object for the current frmit. Fot intra-frame coded objects 

tion method to compute this motion information. Examples the texture decoding block 1674 decodes the color values for 
of motion techniques that can be used in the motion esti- ^ the object and places die reconstructed object in the recon- 

mation block 1634 include the polygon match method stnicted object block 1672. Previously reconstructed objects 

described above, integer pixel motion estimation, etc. The are temporarily stored in object memoiy 1670 and are used 

specific format of the motion information output from the to construct die object for otha frames, 

motion estimation block 1634 can vary depending on the As introduced above, shape infcncation can be used to 

motion estimation method used. For example, the mation determine when motion and/or texture coding can be 

information can include motion vectors for a macroblodc cr skipped for macroblocks and blocks, 

transform coefficients such as the affine transform cocffi- FIG. 40 is a flow diagram illustrating the method imple- 

dents described in detail above. mented in the encoder to skip transparent macroblocks. The 

The motion compensation block 1638 reads the motion shape infonnation generated during the object definition 
data generated in the motion estimation block and the ^ phase of the encoding process can be used to determine 

previously reconstructed object 1636 and conq)utes a pre- which macroblocks and blocks within an object's bounding 

dieted object for the current frame. The encoder finds Hxe rectangle are transparent. Before encoding motion or texture 

difference between the actual object as specified in the input data for a macrobLock, the encoder evaluates the shape data 

1630 and the predicted object as computed in the motion for the current macroblock as shown in step 1700. Decision 

compensation block 1638 to determine the error signal for 45 block 1702 represents the step of determining whether the 

the object current macroblock is entirely transparent from the shape 

Texture coding block 1640 congresses this error signal infonnation. 

for intcrframe coded objects and compresses color values for If the cunent macroblock is transparent, the encoder can 

the object from the input data stream 1630 for intra-frame skip the cunent macroblock without sending any bits to 

coded objects. The feedback path 1642 from the texture 50 indicate that the block has been skipped or any bits used to 

coding block 1640 represents the enor agnal. The encoder represent transparent pixels. 

uses this enor signal along wi& the predicted object from if the cunent macroblock is not entirely transparent, then 

the motion compensation block to compute the previously the encoder codes the macroblock depending on whetiier it 

reconstructed object 1636. is an intra or inter-frame macroblock Decision block 1706 

The texture coding block 1640 codes intra-frame and 55 and the encoding steps 1708, 1710 following it genially 

error signal data for an object using any of a variety of still illustrate the difference in the encoding process for intra and 

image compression tedmiqucs. Example compression tech- interframe macroblocks. For inter-frame blocks, the encoder 

niques indude DCT, wavdet, as well as other conventional encodes both motion estimation infonnation as well as 

image compression methods. Examples of these image texture information. For intra-frame blocks, the encoder 

compression techniques are described in further detail above 60 codes texture and does not necessarily have to encode 

with reference to compression of quantized objects and motion. 

estimated error signals. As illustrated in FIG. 40, the shape information enables 

Thebitstreamof the compressed video sequence indudcs the encoder to identiiy transparent macroblocks and skip 

the shape, motion and texture coded infonnation from the texture encoding and possibly motion encoding for the 

shape coding, motion estimation, and texture coding blocks. 63 macroblock. When it Identifies a transparent macroblock. 

Multiplexer 1644 combines this data into the bitstream and the encoder skips directly to the next macroblock without 

ou^uts it to the buffer 1646. encoding any bits for the cunent macroblock. 
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The shape infoiroatioii can also be used to skip decoding 
operations in the deooder. FIG. 41 is a flow diagram illus- 
tratlDg a method fcs skipping transparent macroblocks in the 
decode. Before decoding motion or texture data, the 
decoder evaluates the shape information for the current 
macroblock as shown in block 1720. Decision block 1722 
reflects that the decoder determines whether a macroblock is 
transparent from the shape information before decoding 
motion or textin'e data. If the current maooblock is entirely 
transparent, the decoder skips the macroblock and proceeds 
directly to the next macroblock as shown in step 1724. 

The type of decoding skipped for the current macroblock 
depends on whether the macroblock represents intra or 
inter-frame coded data. Dedsion block 1726 and the fol- 
lowing steps 1728 and 1730 illustrate the difference in 
decoding intra and inter-frame macroblocks in cases where 
the current macroblock is not entirely transparent If the 
macroblock is not transparent and represents intra-frame 
data, the decoder proceeds to decode texture information for 
the macroblock. If, on the other hand, the current macrob- 
lock is an interframe frame coded macroblock, the decoder 
proceeds to decode motion data and texture data for the 
macroblock. In this case, the texture data represents error 
signals used in motion compensation. 

FIG. 41 generally illustrates how shape information can 
be used to skip transparent maaoblocks. In addition to the 
advantage gained by skipping decoding steps, the decoder is 
further improved since it docs not need to decode any 
overhead bits used to identify "skipped" blocks. 

Shape information can be used to skip transparent por- 
tions of transformation blocks in object-based video coding 
methods where transformation blocks are divided into and 
coded using sub-transformation blocks within each trans- 
formation block. Macroblocks are typically divided into 
smaller blocks so that texture coding can be ^lied on these 
individual blocks within the macroblock. In some 
implementations, motion information can be coded on a 
block by block rather basis as well. Some macroblocks may 
indude one or more transparent blocks. Coding these trans- 
parent blocks adds unnecessary coding an decoding opera- 
tions and adds more bits to the bitstrcam. 

The opportunity to skq> transparent blocks arises for a 
partially covered macroblocks. If a macroblock is entirely 
transparent, the transparent macroblock skipping method 
above can be used to avoid unnecessary coding operations 
and hits in the bitstrcam. On the other hand, if an object 
touches each of the blocks within a macroblock, none of the 
blocks are entirely transparent Therefore, block skipping in 
this context applies to partially covered macrobl(x:ks: mac- 
roblocfks induding at least one transparent block. 

FIG. 35 illustrates an example of a partially transparent 
macroblcKk with at least one transparent block. Macroblock 
1550 indudes three transparent blocks, 1554, 1556, and 
1558, as weU as a nontransparent block 1560. When this 
block is encoded or decoded, texture coding can be skipped 
for each of the transparent blocks 1554, 1556, 1558. In 
addition, if motion information is coded for each block, 
motion coding c^an be skipped for the transparent blocks as 
wdL 

FIG. 42 is a flow diagram illustrating block skipping for 
partially oovaed macroblocks in an encoder. While process- 
ing the current macroblock the encoder evaluates the shape 
for a Hock within this macroblock as shown in step 1750. As 
illustrated by decision block 1752, the encoder evaluates 
whether die block is transparent from the shape information. 
If the cnurent block within the macroblock is entirely 
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transparent, tiie encoder sk^s coding for the block. This 
indudes skipping texture coding for the block and possibly 
motion coding (see stq> 1754). 

In some implementations of object-based video coding 

^ methods, it is sometimes necessary to set certain parameters 
fc3r block (see optional step 1756). Fcet example in one 
inQ)leinentation, the enccxler sets a texture flag indicating 
(hat the texture coef&dents for eac^ pixel in the block are 
zero and also sets a motion flag indicating that the motion 

10 vecrtcff for the transparent block is zero. 

It is necessary to set these flags in this implementation 
because the values of these flags would otherwise be 
undefined, and the undefiDcd stams of these flags could 
actually increase the overhead assodated with the macroh- 
lock that contains the transparent blod^ The overhead could 
be increased because these flags are used to determine when 
decoding of the texture data as seriated with the macroblock 
can be skipped in cases where the texture has not changed 
for a current frame. This form of skipping is different than 

^ skipping a transparent macroblock because it is not dcpcn- 
dent on the extent to which the object covers the macrob- 
lock. Rather, it depends on how the texture data changes 
from frame to frame. In the event that the texture has not 
changed for the macroblock, the decoder does not bave to 

^ decode the texture bec^se it has already constructed the 
same data for a previously reconstructed object. 

The above description of skipping transparent sub- 
transformation blocks in the encoder results when the 
encoder detennines that a block within a partially covered 
transformation block is transparent If the current block is 
not transparent, then the encoder nuist encode texture and 
possibly motion data for the block. Dedsion block 1758 and 
the steps following it (1760 and 1762) represent the diffcr- 
ence in block encoding depending on whether the block is 
intra or inter-frame coded. If the block is an intra-frame 
coded block, the encoder in this inq>lementation only codes 
the tnturc informatioiL On the other hand, if the current 
block is an inter-frame coded block the encoder must ccxte 

^ motion data for the block as wdl. Of course, this only 
applies to implementations where motion data is coded on a 
block by block basis. In cases where only texture data is 
coded on block by block basis, the shape information for 
partially covered macroblocks can only be used to skip 
texture coding for transparent blocks within the macroblock. 

FIG. 43 is a flow diagram illustrating the process for 
transparent block skipping in a decoder. This process is very 
similar to transparent block skipping in the encoder. As the 
deooder decodes the current macroblock, it evaluates the 
shape for the current block as shown In step 1780. Decision 
step 1782 rqiresents the determination whether the current 
block is entirdy transparent If the current block is entirely 
transparent, the decoder skips the block and proceeds to 
decode the next block: 

55 If however, the current block is not entirely transparent 
the decoder proceeds to decode texture data and possibly 
motion data for the block. The decision block 1786 rqire- 
sents the different types of decoding for inter and intra-frame 
coded blcxdcs. For intra-fr^roe coded blocks, the decoder 

M decxHies texture information representing color values for 
the object (see step 1788). For inter-frame coded objects, the 
decoder potentially decodes motion data along witti texture 
data if motion data is coded on a block by block basis within 
the macroblock (see steps 1788 and 1790. 

65 As illustrated in FIGS. 42 and 43, object-based encoders 
and decoders can reduce operations by identifying transpar- 
ent blocks within a transfonnation block from an object's 
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ihBpt infosiaatioii. The methods described above reduce 
texture coding and decoding (^crations sigoiiicantly since 
transparent blocks are not filled with zeros and encoded or 
decoded. Similaiiy, motioD coding at the block level can be 
skqiped as welL 

Block Transparency Status for Intnfraxne Shape 
Coding 

Some object based video coding methods code shape 
information for a frame by using motion estimation and 
compensation on the shape of video objects. Hiis approach 
is similar to motion estimation and coo^asation poformcd 
on texture, excqit that it is performed on shape infonnation 
and residual cnror values arc not coded. This form of shape 
coding is sometimes referred to as inter-frame shape coding. 

When the shape of an c^ject in a sequence of video is 
encoded using motion data^ the decoder must decode motion 
data before it can reconstruct the shape for a frame. Since the 
sh^ information is not available before the motion data, it 
cannot be used to identify transparent blocks within a 
paitially transparent maaoblock. As such, thae is a need fcr 
a method for encoding transparent blocks in partially cov- 
ered macroblocks that arc coded usbg inter-frame shape 
coding. If the status of transparent blocks are encoded in the 
bitstream, the decoder can dicn determine which blocks are 
transparent, even though the shape information is not avail- 
able until after the motion data is decoded. 

One way to encode the status of a partially transparent 
macroblock in these circumstances is to add block transpar- 
ency status information to the bitstream Hie block trans- 
parency status information generally refers to a data bit or 
bits which indicate which blocks within a partially covered 
macroblock are transparent 

The block transparency status infonnation adds additional 
hits to the bitstream, and therefore, should only be used in 
cases where it is necessary to determine whether motion 
decoding should be pcrfoimed for blocks within a macrob- 
lock. In one implementation of this method, the block 
transparency status information is encoded in block trans- 
parency status bits (BTS). We generally refer to this block 
transparency status data as block transparency status flags. 
The BTS data is only present when the following tfiree 
criteria arc satisfied: 

1. Motion prediction/compensation is used to code an 
object In other words, the object is inter-frame coded, and 
specifically, the shape is intcrframe coded 

If shape is not coded using motion compensation, the 
shape can be decoded before motion data is decoded and 
then used to identify transparent blocks within a partially 
covered macroblock. 

2. The macroblock at issue is a partially covered (Le., 
partially transparent ). This information can be determined 
on the encoder side by looking at the shape infonnation and 
determining from the shape information whether an object 
overlaps one or more, but not all, of the blocks in a 
macroblock. 

3. The macroblock is encoded using more than one 
motion vector (e.g., two to four motion vectors, each cor- 
re5ponding to a block within the macroblock). If there is no 
motion vector for a macroblock, then there is no need fcr 
BTS since there will be no motion decoding for the mac- 
roblock anyway. If there is only one motion vector for the 
macroblock, it needs to be decoded and there is no addlUonal 
benefit for using BTS data. 

The in^>lementation of the BTS requires support in the 
encoder and decoder Tlie encoder must evaluate when the 



8,789 

50 

BTS should be added to the con^ssed bitstream based on 
the above criteria. To summarize, the encoder adds the BTS 
to the bitstream in cases where shape is intcr-framc coded, 
the macroblock is partially transparent and there are at least 

5 two motion vectors for the macroblock. 

Since the encoder selects the type of coding for each 
object and macroblock, it knows whether it has coded shape 
using motion conq)ensation for a given macroblock- To code 
sh^e using motion coir^>ensation, the encoder performs 

10 motion estimation on the texture data and the output of the 
motion estimation step process is motion data (e.g. motion 
vectors) defining how shape or texture changes between 
different frames. Specifically, in one implementation for 
example, the encoder performs motion estimation on the 

15 luminance data to coit^nite the motion vector or vectors for 
a maaobiock. Tlie motion vectors can also be computed 
from alpha data. 

The encoder applies the motion vector or vectors to a 
shape for a first frame to find the predicted shape for a 

^ second frame. It then con^wtes the enor between the 
predicted shape and the actual shape for the second frame. 
If the cnor is within a predefined tolo-ance, the encoder does 
not need to send the shape for the second frame, but rather, 
it encodes the shape using the motion data. Conversely, if the 

^ enor exceeds the tolerance, the encoder codes the shape for 
the second frame and places it in the bitstreanL 

In one specific implementation of inter-frume shape 
coding, the encoder performs motion condensation on ttie 

^ alpha block, and then, before computing the prediction error, 
clamps the compensated alpha data for a block by rounding 
the values to 0 or 255. The encoder computes the prediction 
error for a 16x16 ai^ba block by dividing the block into 
sub-blocks, computing the summation of absolute prediction 
enor for eadi sub-block, and ttien determining whether each 
absolute prediction error is less than the predefined toler- 
ance. In addition to meeting this error criteria, there are three 
other critoia for using inter-frame alpha coding: the com- 
pensated alpha block must not be all zeros, the conq^ensated 
alpha block and the actual block must not be all 255, and the 

^ YUV texture data for the maaobiock must be inter-frame 
coded. 

The encoder can determine whether a macroblock is 
partially transparent by evaluating the shape information for 
the macroblock as e:q>lained previously. Information about 
the alpha data in a macroblock or block is sometimes 
encoded using a special code. For example, in one specific 
in^lementation, a code called the first_MMR_codc is used 
to indicate whether alpha data exists, and indicates cases 

^ where alpha values in a block are all 255 (opaque), or all 
zero (transparent), etc. 

The encoder knows the number of motion vectors used for 
a given macroblock because it controls motion estimation 
and compensation. The mode of die motion coding used in 

55 the coder is typically specified for a macroblock. For 
exaiiq)le, if the motion coding is set to advanced mode, 
meaning that there are 2-4 motion vectors for a macroblock, 
then the macroblock is coded to indicate this advanced 
motion coding mode. 

60 If the above three conditions are detected in the encoder, 
the encoder sets the BTS in the bitstream for a macroblock. 
In one implementation, the BTS is a four bit number where 
each bit indicates whether a corresponding block in the 
macroblock is totally tranparent (zero), or otherwise (one). 

63 On the decoder side, the decoder reads the BTS to 
determine whether it can skip decoding motion data for one 
or more of the blocks in a macroblock. If it can ski^ motion 
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decoding, it moves directly to the next block aad only Reducing the Overhead of the Coded Block Pattern 

decodes motion vector data for blocks for which the corre- for Texture 

spending BTS bit is set Iq gome object based video coding methods, the encoder 

An example of the macroblock structure is as follows: places a code called the Coded Block Pattern (GBP) in the 



Gnt-MXfR-jcode 



COD MCB CBFY DQUA BTS MVD MVPi MVD3 MVD4 



CR 


aO_jcok)r 


VLCJbiDaiy 


RLB/ULB 


CODA 


CBPA 


Alpha31ock 


Block Data 



The codes and data in this example structure are defined 
as follows: 

COD— A bit indicating that the maaoblock is coded. COD 
is only present for macxoblocks of objects that are coded 
using motion predictioo and coiiq)ensation. 
MCBPC— A variable length code word giving information 
about the type of coding used for this macroblock and also 
providing the Coded Block Pattern for Chrominance 
(CBPC). 

CBFY— The Coded Block Pattern for luminance, Y. This 
code is used to specify whether the transform cocfiScnts for 
luminance within a block are all zero. 
DQUANT — A code used to indicate a change in the quan- 
tizer for an object. 

BTS — The block transparency status bits. A four bit code 
word indicating whether each block in a partially transparent 
macroblock is entirely transparent 
MVD — Motion vector information provided for intcr-frame 
coded macroblocks. It includes a variable length code word 
for the horizontal con^nent and a variable length code 
word for the vertical con^nent 

MVDj, MVDg, and MVD4— Motion vector data present in 
advanced prediction mode. 

CR — This refers to a conversion ratio used in shape coding. 
Shape information for an object can be size converted for 
rate control and rate reduction. The conversion ratio can be» 
for example, 1 (for the original size), ¥1 and V*, 
aO_CQlor — One bit code indicating the color of the first 
pixel in a macroblock. 

VljC_binary — ^Variable length code for binary sh^ infor- 
mation. 

RLB/ULB — Residual Length of binary shape infonnation/ 
unchanged length of tnnary shape iofoimation. 

The codes CODA, CBPA, and "Alpha Block Data" are a 
portion of the bitstream representing the encoding of alpha 
data for a macroblock. 

CODA— This is a single bit indicating whether all of the 
values in the alpha macroblock are 255 (the macroblock is 
opaque). 

CBPA— This code is the Coded Block Pattern for Alpha. 
This is the same as CBFY. The CBPA bit is set to zero for 
macroblocks and blocks with all zero ali^a values. 
Alpha Block Data— These are the alpha values for a mac- 
roblock. Note that no data needs to be sent in this portion if 
the macroblock is opaque or is entirely transparent 
Block Data — This is the rest of the data for the blocks in the 
macroblock (e.g., the texture data). 

Note that in this exan^lc, the BTS flags arc provided 
before the motion vector data in the macroblock structure. 
This enables the decoder to determine whether it can skip 
motion decoding for one of the blocks in the macroblocks 
(die motion vectors correspond to the Y sub-transformation 
blocks for the macroblock). Thus, if a block in a partially 
transparent macroblock is entirely transparent, the decoder 
skips ttit step of decoding motion data for that block. 



bitstream to indicate the coded blodc pattern for individual 
15 blocks (sub-transformation blocks) in a macroblock. 

For instance, in the macroblock structure set forth above, 
there are six blocks in a macroblock: four for luminance(Y) 
and 2 for chrominance (Q. In this case, there are two kinds 
of Coded Block Patterns (CBP): CBPY, the coded block 
^ pattern for hmjinancc, and CPBC, the coded block pattern 
for du'ominance. 

In cases where there are one or nK>re transparent blocks 
within a partially transparent macroblock, the overhead 

25 associated with coded block pattern for luminance can be 
reduced t>ecause there is no need to code CBPY data for a 
transparent block. The overhead for coded block pattern data 
for luminance can be reduced because there is normally 
coded block pattern data for each of the 4 blocks, whereas 

3Q the coded block pattern data for a chrominance block is the 
same as the maaoblock it belongs to. The amount of 
overhead for CBPY can be reduced since it no longer 
requires 4 bits for each MB if there are one or more 
transparent blocks in the macroblock. 

35 Rather than transmit a bit for each block, the bitstream 
only needs to include CBCY data for non-^nsparent blocks 
in a partially transparent macroblock. To reduce the number 
of UXs needed to encode (TBPY, therefore, the encoder 
determines whether any of the blocks in a partially trans- 

40 parent macroblock are entirely transparent If at least one of 
the blocks is entirely transparent, then only tiie non- 
transparent blocks are coded. 

Only one bit is needed for each non-transparent block to 
indicate whether the transform coefficients are all zero. In 
one specific implementation, ftese bits corresponding to 
non-transparent blocks are converted into a variable length 
code according to whether the texture in the macroblock is 
an intra^rame coded block (I) or is a predicted type block 
(Le., coded using motion estimation/compensation). 

50 

The encoder uses a variable length code (VLC) table to 
determine which VLC to use for transparent blocks. In the 
tables below, the value represents the block is coded, and 
the value **0** indicates that the block is not coded. Bdow, 

55 we list a specific implementation of the VLC tables in cases 
where there arc 2, 3, 4 non-transparent macroblocks. In 
cases where only one block in the partially transparent 
macroblock is non-transparent, no table is needed since only 
one CBPY bit is sent 

60 1 CBFY: no VLC table is needed. 
2 CBFY: 



CBPY (I) CBPY (P) VLC code number of bits 



65 11 00 1 1 

10 01 01 2 
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-contiiiucd non-zero bit Having at least one non-zm bit facilitates 

enor cfaeddng of (he bitstream. 

While we have desoibcd block skipping with reference to 
specific encoder and decoder structures and methods, it is 
inq}oitant to cn^hasizc that the invention can be applied in 
a variety of object-based video coding systems. The specific 
coding methods employed are not critical to the invention 
and other shape, and motion coding techniques can be used 
besides the specific coding tediniques described and illus- 
trated in this application. The object-based coding tech- 
niques can be implemented in software or hardware coders/ 
decoders <x systems using a combination of hardware and 
software. 

In view of the many possible embodiments to which the 
principles of our invention may be applied, it should be 
recognized that the illustrated embodiments are only 
examples of the invention and should not be taken as a 
limitation on the scope of the invention. Rather, the scope of 
the invention is defined by the following claims. We there- 
fore claim as our invention all that comes within the scope 
and spirit of these claims. 
We claim: 

1. In an object-based video coding method, a method for 
23 reducing coding overhead comprising: 

separately encoding video objects in a sequence of video 
frames including: 

separately encoding shape for each of the objects, 
separately encoding texture for each of the objects, and 
30 while coding texture for a first object, evaluating the 
shape of the first object to determine whether a 
transformation block in the first object is transparent 
based on the shape for the first object, and if so, then 
skipping texture coding for the transformation block; 
35 and 

separately decoding video objects in the sequence of 
video fiames including: 

separately decoding shape for each of the objects, 
separately decoding texture for each of the objects, and 
while decoding texture for a first object, evaluating 
whether a transformation block in the first object is 
transparent based on the shape of the first object, and 
if so, then skipping texture decoding for the trans- 
formation Uock. 
^ 2. The method of claim 1 wherein the steps of evaluating 
the shape are performed repeatedly to identify transparent 
transformation blocks within the first object and other 
objects in the sequence of video frames. 

3. The method of claim 1 wherein the texture of the first 
object is intra-frame coded. 

4. The method of claim 1 wherein the texture of the first 
The last table used for cases where all four blocks are obiaA is intcr-framc coded. 

non-transparent is not new. It is the standard table currendy method of claim 1 wherein the step of separately 

proposed in MPEG'4. "Die coding approach of CBFY for encoding video objects indudes the following steps: 

cases where there is at least one transparent block arc new separately coding motion data for eadi of the objects, 

and provide a ^gnificant reduction in the number of Wu ^^^^ f^, ^ ^^^^ evaluating 

needed to encode partia^ti^parent m^^^^^ whether the transformation block is transparent, and 5 

^Jn^fcT "^^^^ *^ so, then skipping motion coding for the ^formation 

ary of an object ... -t-i-^ 

The CBPY bits used for partially transparent macroblocks ^ ^ 

provide a significant reduction in the number of bits needed wherein the step of sq)aratcly decoding the video 

to encode the Coded Block Pattern relative to the table used objects includes the following steps: 

to encode four CBPY bits. The VLC bits are selected such separately coding motion data for each of the objects, 

tiiat the shortest codes are used for Coded Block Patterns while decoding the motion data for a first object, 

that tend to occur most fi-equently for Intra-&an^ and 65 evaluating whether the transfomoation block is 

Inter-frame (Predicted) macroblocks. While not a transparent, and if so, then skipping motion decoding 

requirement, the VLC codes should include at least one for the transformation block: 
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6. The method of daim 1 fuitbcr including: 

while coding texture for a transfonnation block partially 
covered by the first object, evaluating the shape of the 
first object to dctennine whether a subtransfonnation 
block in the partially covered transformatioD block is 
transparent based on the shape of the first object and if 
so, then skipping texture coding for the sub- 
transformation block. 

7. The method of daim 1 further induding: 

while coding motion for a transformation block partially 
covered by the first object, evaluating the shape of the 
first object to dctennine whether a subtransfermation 
block in the partially covered transfcamatioa block is 
transparent based on the shape of the first object, and if 
so« then skipping motion coding for the sub- 
transformation block. 

8. The method of daim 1 further induding: 

while coding texture and motion for a transformation 
block partially covered by the first object, evaluating 
the shape of the first dbjeci to determine whether a 
sub-transformation block in the partially covered trans- 
formation block is transparent based on the shape of the 
first object, and if 5o« th^ skipping texture and motion 
coding for the sub- transformation block. 

9. The method of daim S further induding: 

if the sub-transformation block is transparent, encoding a 
motion flag associated with the sub-transformation 
block indicating that there is no motion data assodated 
with the sub-transformatioQ block, and encoding a 
texture flag assodated with the sub-transformalion 
block indicating that there is no texture data assodated 
with the sub-transformation block; and 

while decoding the partially covered transformation 
block, reading the motion flag and the texture flag 
assodated with sub-transformation blocks of the par- 
tially covered transformation block to determine 
whether the partially covered transformation block 
indudes motion or texture data that requires decoding. 

10. An object-based video coder for coding objects in a 
video sequence into a Hi stream, where the objects con^irise 
portions of video frames in the video sequence and are each 
associated with a bounding rectangle that encloses the 
objects in the video frames, and where the bounding rect- 
angles are divided into transfonnation blocks, the encoder 
comprising: 

a shape encoder for coding shape of the objects in the 
video sequence; 

a motion encoder for computing motion estimation data 
for the objects in the video sequence, for computing 
error values between predicted objects and the objects 
in the video sequence, and for coding the motion 
estimation data for transformation blocks covered by 
the objects; 

a texture encoder for encoding pixels con:q)rising the 
objects for transformation blocks covered by the 
objects, and in communication with the motion encoder 
for encoding the error values foa* the transfarmation 
blocks covered by the objects; wherein the texture 
encoder is operable to read the shape of the objects to 
identify the transparent transformation blocks and oper- 
able to skip the encoding of pixels and error values of 
the transparent transformation blocks; and 

a multiplexor in communication with the shape, texture 
and motion coder for combining encoded shape, 
motion estimation and texture data into the bitstream, 

11. The encoder of claim 10 wherein the motion coder is 
operable to read the shape of the objects to identify trans- 
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parent transformation blocks in the bounding rcxlan^cs and 
is operable to skip the coding of motion estimation data for 
the transparent transformation blocks. 

12. The encoder of daim 10 whcrdn the transformation 
s blocks are divided into sub-transformation blocks, wherein 

the texture coder is operable to read the sh^e information 
of the objects to identify transparent subtransfonnation 
blocks, and wherein the texture coder is operable to skip 
texture coding of the transparent sub-transfonnation blocks. 

13. An object-based video decoder for decoding a bit- 
stream of compressed video objects into objects in a video 
sequence, where ^e objects comprise portions of video 
frames in the video sequence and are each associated with a 
bounding rectangle that cndoses the objects in the video 
frames, and where the bounding rectangles are divided into 
transfarmation blodcs, the decoder comprising: 

a shape decoder for decoding sh^^ of the compressed 
objects in the bitstream; 

a texture decoder for decoding pixels frt>m the oom- 
pressed objects, and for decoding the error values for 
the objects; wherein the texture decoder is operable to 
read the shape of the objects to identify the transparent 
transfarmation blocks and operable to skip the decod- 
ing of pixels and error values of the transparent trans- 
2^ formation blocks 

a motion decoder fcr computing motion estimation data 
for the objects in the video sequence, for computing 
error values between predicted objects and the objects 
in the video sequence, and for coding the motion 
estimation data for transfarmation blocks. 

14. A computer readable medium on which is stored 
software for coding video data, which when executed by a 
computer, perform the steps of: 

s^aratdy encoding video objects in a sequence of video 
35 frames induding: 

sq>arately coding shape for each of the objects, 
separately coding texture for each of the objects, and 
while coding texture for a fint object, evaluating 
whether a transformation block is covered by the 
40 shape of the first object based on the shape for the 

first object, and if not, then skipping texture coding 
for the transformation blodc 

15. In an object-based video coding method, a method for 
reducing coding overhead comprising: 

AS separatdy encoding video objects in a sequence of video 
frames induding: 

encoding shape for each of the objects, including 
performing motion coii^>ensation on at least a first 
partially transparent transfonnation block in a first 
50 object, 

setting a block transparency status flag corresponding 
to an entirely transparent sub-transformation block 
in the partially transparent transformation block, 
encoding motion data fcr at least the first object, and 
55 encoding texture for each of the objects; and 

separately decoding video objects in the sequence of 
video frames induding: 

decoding motion data for at least the first object, 
induding evaluating the block transparency flag, and 
60 skipping motion decoding for the cntirdy transpar- 

ent sub-transformation block; 
decoding shape for each the objects, 
decoding texture for cadi of the objects, and 
reconstructing the sequence of video frames from the 
65 decoded shape, motion and texture data. 

16. The method of daim 15 wherein the step of separatdy 
encoding video objects indudes: 



05/13/2004, EAST Version: 1.4.1 



5,748; 

57 

encoding shape, texture, and motioD vecton in transfor- 
mation blocks for each of a plurality of the objects in 
the video sequence, wherein at least some of &e 
transfcnnatioa blocks are partially transparent trans- 
fonnation blocks; 5 

setting a block transparency status flag associated with 
sub-transformation blocks within the partially transpar- 
ent transformation blodcs to indicate which sub- 
transfonnatioo blocks within the partially transparent 
transfonnation blocks, if any, are entirely transparent; 10 
and 

whacin the step of separately decoding the video objects 
includes: 

evaluating the block transparency status flags for the 
partially transparent transformation blocks; and skip- 
ping decoding of the motion vectors for sub- 
transformation blocks for which a corresponding block 
transparency flag is set 

17. In an object-based video coding method, a method for ^ 
reducing coding overhead comprising: 

separately encoding video objects in a sequence of video 
frames including: 

separately encoding shape for each of the objects, 
separately encoding texture for each of the objects, and 25 
while coding texture for a transformation block par- 
tially covered by a first object, evaluating the shape 
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of the first objett to determine v^liether a sub- 
transformAtion block in the partially covered trans- 
formation block is transparent based on the shape of 
the first object, and if so, then skipping texture 
coding for the sub-transfomation block and encod- 
ing a coded block pattern bit or bits only for non- 
transparent sub-transformation blocks in the partially 
covered transformation block; and 
separately decoding video objects tn the sequence of 
video frames including: 

separately decoding shape for each of the objects, 
separately decoding texture for each of the objects, and 
while decoding texture for a transformation block par- 
tially covered by the first object, evaluating the shape 
of the first object to determine wheAer a sub- 
transfonnatxoQ block in the partially covered trans- 
formation block is transparent based on the shape of 
the first object, and if so, then skipping texture 
decoding for the sub-transformation block. 
18. The mettiod of claim 17 wherein coded block pattern 
bit or bits con^se a variable length code selected firom one 
of three variable length code tables where each of the 
variable length code tables correspond to the number of 
non-transparent sub-transformation blocks in the transfor- 
mation block. 

***** 
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It is certified that error appears in the above-iderttified patent and that said Letters Patent is hereby 
corrected as shown below: 



Column 1, line 21, "a fill-length" should read --a full-length-. 
Column 14, line 58, r'^ y g*^ ^ and by'' should read -r^y gy.and by 

Column 15, line 10, "and yl " should read - and yl". 

Column 18, line 54, "(x^ should read -(x, , 

Column 21, line 31, "(xp^^)" should read "(x,,>^|) --. 

Column 21, line 54, "x[ = ax^ + C" should read -JcJ =aX; +C 

Column 21, line 61, = fl cosflc + sin^ + c " should read 

x' = a cos6ic + sin^ + c-. 

Column 21, line 62, = sinflc + a + cos^ + /" should read 

- ^' = sm^+a + cos^+ / 

Column 25, line 6, "is that is" should read -is that it-. 
Column 25, line 7 "-padded padded** should read - -padded-. 
Column 27, line 11, "component 10" should read -component lo". 
Column 28, line 17, "vidoe" should read -video-. 
Column 29, line 10, between "processes" and "video" please insert 
such as MPEG-1, MPEG-2, and H.26X, the decompression or decoding of the- 



Column 30, line 17, "Chanin Encoding" should read -Chain Encoding- 
Column 32, line 16, "Huffinan" should read -Huffman-. 
Column 32, line 20, "Huffinan" should read -Huffman-. 
Column 32, line 21, "Huffinan" should read -Huffman-. 
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Column 33, line 56, "the defines" should read -that defines-. 

Column 35, line 9, "is stored theD" should read -Is stored In the-. 

Column 36, line 31, "succeeding 30 bitmap- should read -succeeding 

bitmap-. 

Column 36, line 49, "Arbitraty" should read -Arbitrary -. 

Column 38, line 52, 'a provide" should read -and provide-. 

Column 39, line 43, "formalD should read -format-. 

Column 47, line 38, "block rather basis as well" should read -block by 

block basis as well-*. 

Column 47, line 40, "coding an" should read -codir^g and — • 

Column 48, line 64, "1790.- should read -1790).-. 

Column 49, line 51, "The macroblock at issue is a partially covered 
(i.e., partially transparent)." should read -The macroblock at issue is partically covered 
(i.e., partially trjansparent).-. 

In the Claims: 

Column 56, line 24, "blocks" should read -blocks;-. 
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